Information collection system, electronic shelf label, electronic pop advertising, and character information display device

ABSTRACT

The present technology relates to an information collection system, an electronic shelf label, an electronic POP advertising, and a character information display device that allow information with higher accuracy to be collected. An information collection device that collects information associated with a customer or a product is included for each product type. Alternatively, an information processing device that supplies information associated with a product and an information display terminal that displays information associated with a product supplied from the information processing device are included. The present disclosure can be applied, for example, to an electronic shelf label, an image pickup device, a communication device, an information processing device, electronic equipment, a computer, a program, a storage medium, a system, and the like.

TECHNICAL FIELD

The present technology relates to an information collection system, anelectronic shelf label, an electronic POP advertising, and characterinformation display device, and, in particular, to an informationcollection system, an electronic shelf label, an electronic POPadvertising, and a character information display device that are allowedto collect information with higher accuracy.

BACKGROUND ART

Conventionally, there has been considered collecting informationassociated with a product and a customer who purchases the product in astore and the like that sells the product.

For example, there has been considered installing a camera in a vendingmachine to collect information associated with a customer purchasing aproduct (for example, refer to Patent Document 1). Furthermore, therehas been considered providing a camera on a display shelf of a productto collect information associated with a customer around the displayshelf (for example, refer to Patent Document 2).

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2009-042956-   Patent Document 2: Japanese Patent Application Laid-Open No.    2013-144001

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

However, in the methods described in the above patent documents, thenumber of installed cameras is smaller than the number of product typesbeing sold. Therefore, it has been difficult to estimate, with highaccuracy, which product is looked at by a customer.

The present technology has been proposed in consideration of the abovesituations, and an object of the present technology is to collectinformation with higher accuracy.

Solutions to Problems

An information collection system according to an aspect of the presenttechnology is an information collection system that includes aninformation collection device configured to collect informationassociated with a customer or a product for each product type.

The number of the information collection devices installed in locationsin which a product is placed can be larger than or the same as thenumber of product types for which the information is collected.

The number of the information collection devices installed in locationsin which a product is placed is larger than or the same as the number ofproduct types in the information for each product transferred from theinformation collection device to a data accumulation device thataccumulates the information.

The number of the information collection devices installed in locationsin which a product is placed can be larger than or the same as thenumber of product types for which the information is output.

The information collection device can be a network camera that includesan image pickup function and a communication function.

The information collection device can be an electronic shelf label thatincludes an image pickup function and a communication function.

The information collection device can be an electronic point of purchase(POP) advertising that has an image pickup function and a communicationfunction.

The information collection device can be a character information displaydevice that includes an image pickup function and a communicationfunction.

The information collection device can acquire and display informationsupplied from other devices.

The other devices can supply the information associated with a productsupplied from the information collection device corresponding toidentification information of the product to the information collectiondevice, and the information collection device can display the suppliedinformation associated with the product.

An information electronic shelf label according to another aspect of thepresent technology is an electronic shelf label that includes aninformation collection unit that collects information associated with acustomer or a product.

The information collection unit can be an image sensor.

An electronic point of purchase (POP) advertising according to anotheraspect of the present technology is an electronic POP advertising thatincludes an information collection unit that collects informationassociated with a customer or a product.

The information collection unit can be an image sensor.

(Image Sensor)

A character information display device according to another aspect ofthe present technology is a character information display device thatincludes an information collection unit that collects informationassociated with a customer or a product.

The information collection unit can be an image sensor.

According to an aspect of the present technology, an informationcollection device installed for each product type collects informationassociated with a customer or a product.

According to another aspect of the present technology, informationassociated with a customer or a product is collected.

Effects of the Invention

According to the present technology, information can be processed.Furthermore, according to the present technology, information withhigher accuracy can be collected.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram explaining an example of a conventional vendingmachine.

FIG. 2 is a diagram explaining an example of a conventional displayshelf.

FIG. 3 is a diagram explaining an example of conventional gazeestimation.

FIG. 4 is a diagram explaining an example of conventional gazeestimation.

FIG. 5 is a diagram showing a main configuration example of aninformation collection system.

FIG. 6 is a diagram showing another configuration example of theinformation collection system.

FIG. 7 is a diagram showing an example of an information collectiondevice.

FIG. 8 is a block diagram showing a main configuration example of eachdevice of the information collection system.

FIG. 9 is a block diagram showing a main configuration example of theinformation collection device.

FIG. 10 is a diagram explaining an example of information to becollected.

FIG. 11 is a diagram explaining a configuration example of theinformation collection system and an example of a state of informationcollection.

FIG. 12 is a diagram explaining a configuration example of theinformation collection system and an example of a state of informationcollection.

FIG. 13 is a diagram explaining a configuration example of theinformation collection system and an example of a state of informationcollection.

FIG. 14 is a diagram explaining a configuration example of theinformation collection system and an example of a state of informationcollection.

FIG. 15 is a diagram showing an example of a power supply structure.

FIG. 16 is a diagram showing an example of the power supply structure.

FIG. 17 is a diagram showing an example of the power supply structure.

FIG. 18 is a diagram showing an example of the power supply structure.

FIG. 19 is a diagram explaining an example of a state of identifying aproduct type looked at by a customer.

FIG. 20 is a diagram explaining an example of a state of identifying aproduct type looked at by a customer.

FIG. 21 is a diagram explaining an example of a state of identifying thesame person.

FIG. 22 is a diagram explaining an example of a state of identifying thesame person.

FIG. 23 is a diagram explaining an example of a state of identifying thesame person.

FIG. 24 is a diagram explaining an example of a state of identifying aproduct type looked at by a customer.

FIG. 25 is a diagram explaining an example of a state of identifying aproduct type looked at by a customer.

FIG. 26 is a diagram showing a main configuration example of aninformation display system.

FIG. 27 is a diagram showing a main configuration example of aninformation display terminal.

FIG. 28 is a diagram showing a main configuration example of anelectronic shelf label system.

FIG. 29 is a diagram explaining an example of a state of arrangement ofan electronic shelf label.

FIG. 30 is a block diagram showing a main configuration example of theelectronic shelf label.

FIG. 31 is a block diagram showing a main configuration example of animage recognition unit.

FIG. 32 is a block diagram showing a main configuration example of aserver.

FIG. 33 is a flowchart explaining an example of a process of informationacquisition processing.

FIG. 34 is a flowchart explaining an example of a process of imagerecognition processing.

FIG. 35 is a flowchart explaining an example of a process of the imagerecognition processing.

FIG. 36 is a flowchart explaining an example of a process of image dataselection processing.

FIG. 37 is a flowchart explaining an example of a process of the imagedata selection processing.

FIG. 38 is a flowchart explaining an example of a process of the imagedata selection processing.

FIG. 39 is a flowchart explaining an example of a process of dataprovision processing.

FIG. 40 is a flowchart explaining an example of a process of displaycontrol processing.

FIG. 41 is a flowchart explaining an example of a process of the dataprovision processing.

FIG. 42 is a flowchart explaining an example of a process of displaydata acquisition processing.

FIG. 43 is a flowchart explaining an example of a process of theinformation acquisition processing.

FIG. 44 is a flowchart explaining an example of a process of the displaycontrol processing.

FIG. 45 is a flowchart explaining an example of a process of dataacquisition processing.

FIG. 46 is a diagram explaining an example of a state of identifying aface position.

FIG. 47 is a flowchart explaining an example of a process of customerinformation recognition processing.

FIG. 48 is a diagram explaining an example of a state of a moving objectdetection function.

FIG. 49 is a block diagram showing a main configuration example of theelectronic shelf label.

FIG. 50 is a flowchart explaining an example of a process of thecustomer information recognition processing.

FIG. 51 is a flowchart explaining an example of a process of thecustomer information recognition processing.

FIG. 52 is a block diagram showing a main configuration example of theimage recognition unit.

FIG. 53 is a flowchart explaining an example of a process of the imagerecognition processing.

FIG. 54 is a flowchart explaining an example of a process of recognitiondata selection processing.

FIG. 55 is an external view showing a main configuration example of theelectronic shelf label.

FIG. 56 is a diagram explaining an example of a state of picking up animage of a customer.

FIG. 57 is a diagram explaining an example of face recognition within amovable range of a camera.

FIG. 58 is a block diagram showing a main configuration example of theelectronic shelf label.

FIG. 59 is a diagram explaining an example of power supply control.

FIG. 60 is a flowchart explaining an example of a process of the powersupply control processing.

FIG. 61 is a flowchart explaining an example of a process of theinformation acquisition processing.

FIG. 62 is a flowchart continuing FIG. 48 explaining an example of aprocess of the information acquisition processing.

FIG. 63 is a block diagram showing a main configuration example of acomputer.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, a mode for performing the present disclosure (hereinafterreferred to as an embodiment) will be described. Note that descriptionwill be made in the order described below.

1. First embodiment (information collection system, information displaysystem)

2. Second embodiment (electronic shelf label system)

3. Third embodiment (selection by electronic shelf label)

4. Fourth embodiment (first application example: face sizeidentification)

5. Fifth embodiment (second application example: moving objectdetection)

6. Sixth embodiment (third application example: display control)

7. Seventh embodiment (fourth application example: movable image pickupunit)

8. Eighth embodiment (fifth application example: power supply controlunit)

9. Ninth embodiment (sixth application example: control of facedirection detection processing)

10. Tenth embodiment (others)

1. First Embodiment

<Collection of Information of a Customer and a Product>

Conventionally, there has been considered collecting informationassociated with a product and a customer who purchases the product in astore and the like that sells the product. For example, Patent Document1 discloses a vending machine 1, in which a camera 2 is installed, thatpicks up an image of a customer who purchases a product by using thecamera 2, and collects the picked-up image as information associatedwith the customer (FIG. 1).

Furthermore, Patent Document 2 discloses a vending machine 1, in which acamera 11 is provided on each shelf of a display shelf 10 on which aproduct is displayed, that picks up an image of a customer around thedisplay shelf 10 by using the camera 11, and collects the picked-upimage as information associated with the customer (FIG. 2).

However, in a case of the above methods, the number of cameras issmaller than the number of product types. Therefore, it has beendifficult to estimate, with high accuracy, which product is looked at bya customer from a picked-up image. A reason for the above will bedescribed by using FIG. 3. Before that, definition of words used in thedescription of the present application will be described. Normally, aplurality of types of products are arranged on a display shelf ofproducts (in other words, a store shelf), and a plurality of products(individuals) are arranged for each type of products as individuals ofproducts. In the description of the present application, products thatare only different from each other individually and have the samecontent in a case where they are referred to collectively will bereferred to as a “product type”. Furthermore, each of products that aredifferent from each other individually in one product type will bereferred to as a “product individual”. However, for simplification,there is a case where a “product type” is simply referred to as a“product”, and there is also a case where a “product individual” issimply referred to as a “product”. A of FIG. 3 is a diagram showing astate in which, as an example of a conventional technology where thenumber of cameras is smaller than the number of product types, seventypes of products (in other words, seven product types, 21-1 to 21-7)and two cameras (22-1 and 22-2) fewer than the number (seven) of theproduct types are arranged on a display shelf 20 on which products aredisplayed, and two customers (23-1 and 23-2) stand in front of thedisplay shelf 20 and look at the products, when viewed from above theshelf. The camera 22-1 picks up an image of the customer 23-1 positionedin front of itself. Furthermore, the camera 22-2 picks up an image ofthe customer 23-2 positioned in front of itself. At this time, thecustomer 23-1 faces the front (toward the camera 22-1). For this reason,it is comparatively easy to estimate that the customer 23-1 looks in adirection of the product 21-2 based on a picked-up image obtained by thecamera 22-1.

However, since the customer 23-2 looks to the side, a profile of thecustomer 23-2 is shown in a picked-up image obtained by the camera 22-2like a picked-up image 24 shown in B of FIG. 3. In A of FIG. 3, an imageof the customer 23-2 is not picked up by the camera 22-1, and picked uponly by the camera 22-2. For this reason, in order to identify which ofthe products 21-3 to 21-5 described in A of FIG. 3 is looked at by thecustomer 23-2 whose image of a profile is picked up by the camera 22-2,it is necessary to estimate a product to which a gaze of the customer23-2 extends on the basis of a slight difference in directions of theface of the customer 23-2 or a slight difference in positions of a pupilin an eyeball of the customer 23-2. For this reason, identifying aproduct looked at by a customer with high accuracy is accompanied by atechnical difficulty. Besides, even if a product to which a gaze of thecustomer 23-2 extends is estimated, there has not been a means forverifying whether or not a result of the estimation is correct. Underthe above circumstances, in a case of the conventional technology inwhich the number of cameras is smaller than the number of product types,it has been difficult to estimate with high accuracy which product islooked at by a customer, and verify whether or not an estimation resultis correct.

Furthermore, if a plurality of cameras are installed, there has beenpossibility that the same person is shot by a plurality of cameras, and,in such a case, there has been possibility that information associatedwith the same customer is collected in a duplicate manner. A reason forthe above will be described by using FIG. 4. A of FIG. 4 is a diagramshowing a state in which three customers (23-1 to 23-3) stand and lookat products in front of the display shelf 20 on which seven types ofproducts (21-1 to 21-7) and three cameras (22-1 to 22-3) are arrangedwhen viewed from above the shelf. As shown in B of FIG. 4, a picked-upimage 24-1 obtained by the camera 22-1 shows the customer 23-1 and thecustomer 23-2. Furthermore, as shown in C of FIG. 4, a picked-up image24-2 obtained by the camera 22-2 shows the customer 23-1, the customer23-2, and the customer 23-3. Moreover, as shown in D of FIG. 4, apicked-up image 24-3 obtained by the camera 22-3 shows the customer 23-2and the customer 23-3.

When customer information is directly collected from these picked-upimages, customer information for seven customers are obtained where onlythree customers actually exist. That is, for example, there has beenpossibility that wrong information, such as “three customers look at theproduct 21-2”, “two customers look at the product 21-3”, “two customerslook at the product 21-4”, and the like, is collected. Accordingly, ithas been difficult to collect information with high accuracy.

In view of the above, an information collection device that collectsinformation associated with a customer or a product is included for eachproduct type and in the vicinity of each product type. In this manner,the information collection device and a product type can be easilyassociated (in other words, linked), and a customer who is looking in adirection of the information collection device can be easily identifiedas looking at a product type corresponding to the information collectiondevice. Furthermore, since a direction in which a customer faces can beeasily identified with high accuracy, collection of duplicate customerinformation can easily be restricted. Accordingly, information withhigher accuracy can be collected.

<Configuration of Information Collection System>

FIG. 5 is a diagram showing a main configuration example of aninformation collection system that is an embodiment of an informationprocessing system to which the present technology is applied. Aninformation collection system 100 shown in FIG. 5 is a system thatcollects information associated with a product and a customer.

As shown in FIG. 5, the information collection system 100 has shelfdevices 101-1 to 101-M, a relay device 102, and an informationmanagement server 103. The shelf devices 101-1 to 101-M haveconfigurations similar to each other. Hereinafter, the shelf devices101-1 to 101-M will be referred to as the shelf device 101 in a casewhere the shelf devices 101-1 to 101-M do not need to be distinguishedfrom each other.

The shelf device 101 has a plurality of product display shelves 121 thatare arranged in a line in a horizontal direction. Products 131 aredisplayed, for example, for each product type on each of the productdisplay shelves 121. Furthermore, an information collection device 122that collects information associated with a product, a customer, and thelike is installed for each product type in the vicinity of the product131 on the product display shelf 121. Moreover, the shelf device 101 hasa local server 123.

The information collection device 122 has a function of collectinginformation, and collects information associated with a product, acustomer, and the like by using the function. The information collectiondevice 122 is installed for each product type of the product displayshelf 121 as well as in the vicinity of each product type. In otherwords, the information collection device 122 is installed in thevicinity of product types (the products 131) different from each otheron the product display shelf 121. In this manner, for example, theinformation collection device 122 is associated (in other words, linked)with a product type (the product 131) by this positional relationship.That is, the information collection device 122 is associated with aproduct type (the product 131) positioned in the vicinity of the owndevice. Moreover, the configuration may be such that informationassociated with the product type (product 131) in the vicinity has beenregistered in the information collection device 122. The informationcollection device 122 collects information associated with a customerfor a product type associate with the device itself (for example, acustomer who shows an interest in the product type). That is, each ofthe information collection devices 122 is associated with a product typedifferent from one another, and collects information associated with acustomer for a product type different from one another.

Furthermore, the information collection device 122 has a communicationfunction, and can perform wired communication or radio communication, orboth, with the local server 123 by the communication function. Forexample, the information collection device 122 can supply informationcollected by the device itself to the local server 123. Furthermore, forexample, the information collection device 122 can also acquireinformation from the local server 123.

Note that the information collection device 122 may have an informationprocessing function. For example, the information collection device 122may process information collected by the device itself, and supply theprocessed information to the local server 123. Furthermore, for example,the information collection device 122 may process information acquiredfrom the local server 123.

The local server 123 has a communication function. The local server 123can perform wired communication or radio communication, or both, witheach of the information collection devices 122 of the shelf device 101by the communication function. Furthermore, the local server 123 canperform wired communication or radio communication, or both, with therelay device 102 by the communication function. For example, the localserver 123 can acquire information collected by each of the informationcollection devices 122 of the shelf device 101, and supply theinformation to the relay device 102. Furthermore, for example, the localserver 123 can acquire information from the relay device 102, and supplythe information to each of the information collection devices 122 of theshelf device 101.

Note that the local server 123 may have an information processingfunction. For example, the local server 123 may process informationcollected by each of the information collection devices 122 of the shelfdevice 101, and supply the processed information to the relay device102. Furthermore, for example, the local server 123 may processinformation acquired from the relay device 102, and supply the processedinformation to each of the information collection devices 122 of theshelf device 101.

The relay device 102 has a communication function. The relay device 102can perform wired communication or radio communication, or both, with(the local server 123 of) each of the shelf devices 101 by thecommunication function. Furthermore, the relay device 102 can performwired communication or radio communication, or both, with theinformation management server 103 by the communication function. Forexample, the relay device 102 can acquire information that each of theinformation collection devices 122 collects from (the local server 123of) each of the shelf devices 101, and supply the information to theinformation management server 103. Furthermore, for example, the relaydevice 102 can also acquire information from the information managementserver 103, and supply the information to (the local server 123 of) eachof the shelf devices 101.

Note that the relay device 102 may have an information processingfunction. For example, the relay device 102 may process informationcollected by the information collection devices 122 that is acquiredfrom (the local server 123 of) each of the shelf devices 101, and supplythe processed information to the information management server 103.Furthermore, for example, the relay device 102 may process informationacquired from the information management server 103, and supply theprocessed information to (the local server 123 of) each of the shelfdevices 101.

The information management server 103 has a communication function. Theinformation management server 103 can perform wired communication orradio communication, or both, with the relay device 102 by thecommunication function. For example, the information management server103 can acquire information collected by the information collectiondevices 122 from the relay device 102. Furthermore, for example, theinformation management server 103 can also supply optional informationto the relay device 102.

Furthermore, the information management server 103 has an informationprocessing function and an information storing function. For example,the information management server 103 can process and store informationacquired from the relay device 102. Moreover, the information managementserver 103 is connected to a personal computer (PC) 111, an externalstorage device 112, a display device 113, and a printer 114 asexternally connected electronic equipment, and can also performprocessing by using these pieces of the externally connected electronicequipment. For example, the information management server 103 can causethe personal computer 111 to process information, the external storagedevice 112 to store information, the display device 113 to display animage, and the printer 114 to print out information.

Note that a configuration of the information collection system 100 isoptional, and is not limited to the example of FIG. 5. For example, asshown in FIG. 6, the configuration may be such that the informationcollection devices 122 is provided only for a product type for whichinformation needs to be collected (that is, only for a product type aspart of a group of products displayed on the product display shelf 121).Also in a case where a plurality of product types for which informationneeds to be collected exist, the information collection devices 122 isinstalled for each of the product types.

Note that the configuration may be such that, while the informationcollection system 100 includes the information collection devices 122that are larger in number than product types for which information needsto be collected, information output from the information collectionsystem 100 is narrowed down only to information for the product typesfor which information needs to be collected and output to the outside ofthe information collection system 100.

For example, the configuration may be such that (1) the informationcollection system 100, in which the information collection device 122 isarranged for each of product types that are larger in number thanproduct types for which information needs to be collected, transmitsimage data obtained by picking up an image of product types larger innumber than product types for which information needs to be collected oranalysis result data after image analysis to the information managementserver 103 via the local server 123 and the relay device 102 from theinformation collection devices 122, and the information managementserver 103 performs processing for narrowing down information to beoutput to the outside of the information collection system 100 only toinformation associated with product types for which information needs tobe output, and outputs the narrowed-down information to the outside ofthe information collection system 100.

Furthermore, the configuration may alternatively be such that (2) theinformation collection system 100, in which the information collectiondevice 122 is arranged for each of product types that are larger innumber than product types for which information needs to be collected,transmits image data obtained by picking up an image of product typeslarger in number than product types for which information needs to becollected or analysis result data after image analysis to the relaydevice 102 via the local server 123 from the information collectiondevices 122, and the relay device 102 performs processing for narrowingdown information to be output to the outside of the informationcollection system 100 only to information associated with product typesfor which information needs to be output, and outputs the narrowed-downinformation to the outside of the information collection system 100 viathe information management server 103.

Furthermore, the configuration may alternatively be such that (3) theinformation collection system 100, in which the information collectiondevice 122 is arranged for each of product types that are larger innumber than product types for which information needs to be collected,transmits image data obtained by picking up an image of product typeslarger in number than product types for which information needs to becollected or analysis result data after image analysis to the localserver 123 from the information collection devices 122, and the localserver 123 performs processing for narrowing down information to beoutput to the outside of the information collection system 100 only toinformation associated with product types for which information needs tobe output, and outputs the narrowed-down information to the outside ofthe information collection system 100 via the relay device 102 and theinformation management server 103.

Furthermore, the configuration may alternatively be such that (4) theinformation collection system 100 includes the information collectiondevice 122 arranged for each of product types that are larger in numberthan product types for which information needs to be collected, and eachof the information collection devices 122 performs processing fornarrowing down information to be transmitted from the informationcollection devices 122 to the local server 123 only to informationassociated with product types that need to be output to the outside ofthe information collection system 100, and outputs the narrowed-downinformation to the outside of the information collection system 100 viathe local server 123, the relay device 102, and the informationmanagement server 103.

That is, the number of the information collection devices 122 thatcollect information associated with a customer and a product may belarger than or the same as the number of product types for whichinformation associated with a customer is collected, may be larger thanor the same as the number of product types in information for eachproduct transferred to the information management server 103, or may belarger than or the same as the number of product types for whichinformation associated with a customer is output.

In the above manner, an amount of information to be handled can berestricted. Therefore, increase in a load of a variety of types ofprocessing relating to the information collection can be restricted.Moreover, increase in cost can also be restricted.

<Information Collection Device>

The information collection devices 122 shown in FIGS. 5 and 6 may be anydevice, and information to be collected may be any information as longas the information is associated with a product or a customer. Forexample, the configuration may be such that the information collectiondevice 122 picks up an image of a product or a customer, and an obtainedpicked-up image or information obtained on the basis of the picked-upimage is collected as information associated with a product or acustomer. Furthermore, the configuration may be such that sensing isperformed for a product or a customer by using some sensor, and anobtained sensor output or information obtained on the basis of thesensor output is collected as information associated with a product or acustomer. Furthermore, the configuration may be such that theinformation collection device 122 collects a plurality of types ofinformation (for example, a picked-up image and a sensor output) asinformation associated with a product or a customer.

For example, the information collection device 122 may be configured asa network camera 140 as shown in A of FIG. 7. The network camera 140 hasan image sensor 141 and a ranging sensor 142. For example, the imagesensor 141 is an image sensor using a charge coupled device (CCD) and acomplementary metal oxide semiconductor (CMOS), and picks up an image ofan object and generates (data of) a picked-up image of the object. Notethat the image sensor 141 may have an optical mechanism, such as a lens,a diaphragm, and the like. The ranging sensor 142 is a sensor thatmeasures a distance to an object. The specifications of the rangingsensor 142 is optional, and the ranging sensor 142 may measure adistance by any method. For example, the ranging sensor 142 may measurea distance by using an infrared ray. Furthermore, the network camera 140has a communication function (not shown).

For example, the network camera 140 can pick up an image of a customerby using the image sensor 141, measure a distance to the customer byusing the ranging sensor 142, and supply these pieces of information (apicked-up image, a distance measurement result, and the like) asinformation associated with the customer to the local server 123 by thecommunication function. Furthermore, for example, the network camera 140can pick up an image of an external form of an associated product, abarcode, a character describing a product name, and the like by usingthe image sensor 141, and supply the picked-up image as informationassociated with the product to the local server 123 by the communicationfunction.

Furthermore, for example, the information collection devices 122 mayhave a function of outputting information. For example, the informationcollection device 122 may have a function of displaying a character, animage, and the like. Furthermore, the information collection device 122may have a function of outputting voice, vibration, and the like. As amatter of course, the information collection device 122 may beconfigured to be able to output a plurality of types of information (forexample, an image, a voice, and the like). Note that the information tobe output is optional. For example, the information may be informationassociated with a product or a customer, or information that is notassociated with a product or a customer.

For example, the information collection device 122 may be configured asan electronic shelf label 150 as shown in B of FIG. 7. The electronicshelf label 150 is a device that displays information associated with aproduct, such as a price of a product, and the like. As shown in B ofFIG. 7, the electronic shelf label 150 has an image sensor 151, aranging sensor 152, and a monitor 153. The image sensor 151 is a sensorbasically similar to the image sensor 141, and picks up an image of anobject to obtain a picked-up image. The ranging sensor 152 is a sensorbasically similar to the ranging sensor 142, and measures a distance byusing, for example, an infrared ray and the like. The monitor 153 is adisplay device that displays character information and the like.Furthermore, the electronic shelf label 150 has a communication function(not shown).

For example, the electronic shelf label 150 can pick up an image of acustomer by using the image sensor 151, measure a distance to thecustomer by using the ranging sensor 152, and supply these pieces ofinformation (a picked-up image, a distance measurement result, and thelike) as information associated with the customer to the local server123 and the like by the communication function. Furthermore, forexample, the electronic shelf label 150 can picks up an image of anexternal form of an associated product, a barcode, a characterdescribing a product name, and the like by using the image sensor 151,and supply the picked-up image as information associated with theproduct to the local server 123 and the like by the communicationfunction.

Furthermore, for example, the electronic shelf label 150 can acquireinformation associated with a product from the local server 123 and thelike by the communication function, and causes the monitor 153 todisplay the information associated with the product as characterinformation and the like. A content of the information associated with aproduct is optional. For example, information of a product name, aprice, a content amount, and the like may be included in the informationassociated with a product.

Furthermore, for example, the information collection devices 122 may beconfigured as an electronic point of purchase (POP) advertising 160 asshown in C of FIG. 7. The electronic POP advertising 160 is a devicethat displays information associated with a product, such as anadvertisement of a product, and the like. As shown in C of FIG. 7, theelectronic POP advertising 160 has an image sensor 161, a ranging sensor162, a monitor 163, and speakers 164-1 and 164-2. The image sensor 161is a sensor basically similar to the image sensor 141, and picks up animage of an object to obtain a picked-up image. The ranging sensor 162is a sensor basically similar to the ranging sensor 142, and measures adistance by using, for example, an infrared ray and the like. Themonitor 163 is a display device that displays optional imageinformation, such as video, a still image, or the like. Furthermore, themonitor 163 may display optional image information, such as video, astill image, or the like in an area in part of a screen, and displaycharacter information in an area on the screen different from the abovearea. Alternatively, the monitor 163 may display only characterinformation. Note that, needless to say, image information displayed bythe monitor 163 may include character information. The speakers 164-1and 164-2 are voice output devices that output voice. Note that thespeakers 164-1 and 164-2 will be referred to as the speaker 164 in acase where the speakers do not need to be distinguished from each other.Furthermore, the electronic POP advertising 160 has a communicationfunction (not shown).

For example, the electronic POP advertising 160 can pick up an image ofa customer by using the image sensor 161, measure a distance to thecustomer by using the ranging sensor 162, and supply these pieces ofinformation (a picked-up image, a distance measurement result, and thelike) as information associated with the customer to the local server123 and the like by the communication function. Furthermore, forexample, the electronic POP advertising 160 can pick up an image of anexternal form of an associated product, a barcode, a characterdescribing a product name, and the like by using the image sensor 161,and supply the picked-up image as information associated with theproduct to the local server 123 and the like by the communicationfunction.

Furthermore, for example, the electronic POP advertising 160 can acquireinformation associated with a product from the local server 123 and thelike by the communication function, and causes the monitor 163 todisplay the information associated with the product as moving images, astatic image, a character, or the like. A content of the informationassociated with a product is optional. For example, information of aproduct name, a price, a content amount, and the like may be included inthe information associated with a product. Furthermore, for example, theinformation may be advertisement video and the like of a product.Furthermore, the electronic POP advertising 160 can cause informationassociated with a product acquired from the local server 123 to beoutput as voice from the speaker 164.

Note that, although illustration is omitted, for example, theinformation collection device 122 may be configured as a characterinformation display device, such as an electric bulletin board. Thiselectric bulletin board is a device that displays optional characterinformation, and can display, for example, character information, suchas an advertising message of a product, a guidance for a selling area,and the like. In this case as well, the information collection device122 further has an information collection function, such as an imagesensor, a ranging sensor, and the like, as similar to the cases of theelectronic shelf label 150 and the electronic POP advertising 160described above, and collects information associated with a product anda customer. Furthermore, the information collection device 122 also hasa communication function (not shown), and can supply collectedinformation to the local server 123 and the like, and acquire optionalinformation from the local server 123 and the like. Note that, in thiscase as well, like the case of the electronic POP advertising 160, theinformation collection device 122 may be configured to have a functionof outputting information other than an image, for example, a speakerand the like.

<Configuration of Each Device>

FIG. 8 shows a main configuration example of the local server 123 andthe information management server 103 of the information collectionsystem 100. In a case of the example of FIG. 8, the local server 123 hasa communication unit 171 and an image analysis unit 172. Thecommunication unit 171 performs processing for performing acommunication function of the local server 123 described above. In otherwords, the communication unit 171 performs wired communication or radiocommunication, or both, with each of the information collection devices122 (the information collection devices 122-1 to 122-N) of the shelfdevice 101 and the relay device 102. Furthermore, the image analysisunit 172 analyzes a picked-up image included in information that thecommunication unit 171 acquires from the information collection devices122, and supplies a result of the image analysis to the communicationunit 171. For example, the image analysis unit 172 identifies whichproduct is looked at by a customer by analyzing a picked-up image, andestimates attributes (gender, age, and the like) of a customer. Thecommunication unit 171 supplies information acquired from theinformation collection devices 122 and a result of image analysisperformed by the image analysis unit 172 to the relay device 102.

Furthermore, in a case of the example of FIG. 8, the informationmanagement server 103 has a communication unit 181, an informationcontrol unit 182, and an internal storage device 183. The communicationunit 181 performs processing for performing a communication function ofthe information management server 103 described above. The informationcontrol unit 182 controls other processing units, externally connecteddevices, and the like, and processes information. For example, theinformation control unit 182 performs processing, such as reading andwriting information received or transmitted by the communication unit181 in the internal storage device 183 and the external storage device112, outputting the information to the display device 113 and theprinter 114, providing or receiving the information to or from thepersonal computer 111, and the like.

FIG. 9 shows a main configuration example of the information collectiondevice 122. A of FIG. 9 is a block diagram showing a configurationexample in a case where the information collection device 122 isconfigured as the network camera 140. As shown in A of FIG. 9, thenetwork camera 140 has an image pickup unit 201, a ranging unit 202, anda communication unit 203. The image pickup unit 201 has the image sensor141, and performs processing relating to image pickup of an object. Theranging unit 202 has the ranging sensor 142, and performs processingrelating to measurement of a distance to an object. The communicationunit 203 performs processing for performing the communication functionof the network camera 140 described above. For example, thecommunication unit 203 supplies data, such as a picked-up image obtainedby the image pickup unit 201, a distance measurement result obtained bythe ranging unit 202, and the like as information associated with acustomer and a product to the local server 123. Note that the networkcamera 140 may have a storage unit 204 that stores electric power, sothat each processing unit is driven by using power of the storage unit204. As a matter of course, the configuration may be such that thestorage unit 204 is omitted, and each processing unit is driven by usingpower supplied from an external power supply.

B of FIG. 9 is a block diagram showing a configuration example in a casewhere the information collection device 122 is configured as theelectronic shelf label 150. As shown in B of FIG. 9, the electronicshelf label 150 has an image pickup unit 211, a ranging unit 212, adisplay unit 213, and a communication unit 214. The image pickup unit211 has the image sensor 151, and performs processing relating to imagepickup of an object. The ranging unit 212 has the ranging sensor 152,and performs processing relating to measurement of a distance to anobject. The display unit 213 has the monitor 153, and performsprocessing relating to display of character information relating to aproduct. The communication unit 214 performs processing for performingthe communication function of the electronic shelf label 150 describedabove. For example, the communication unit 214 supplies data, such as apicked-up image obtained by the image pickup unit 211, a distancemeasurement result obtained by the ranging unit 212, and the like asinformation associated with a customer and a product to the local server123. Furthermore, the communication unit 214 acquires, for example,character information relating to a product from the local server 123,and supplies the character information to the display unit 213. Notethat the electronic shelf label 150 may have a storage unit 215 thatstores electric power, so that each processing unit is driven by usingpower of the storage unit 215. As a matter of course, the configurationmay be such that the storage unit 215 is omitted, and each processingunit is driven by using power supplied from an external power supply.

C of FIG. 9 is a block diagram showing a configuration example in a casewhere the information collection device 122 is configured as theelectronic POP advertising 160. As shown in C of FIG. 9, the electronicPOP advertising 160 has an image pickup unit 221, a ranging unit 222, adisplay unit 223, a voice output unit 224, and a communication unit 225.The image pickup unit 221 has the image sensor 161, and performsprocessing relating to image pickup of an object. The ranging unit 222has the ranging sensor 162, and performs processing relating tomeasurement of a distance to an object. The display unit 223 has themonitor 163, and performs processing relating to display of characterinformation relating to a product. The voice output unit 224 has thespeaker 164, and performs processing relating to voice output. Thecommunication unit 225 performs processing for performing acommunication function of the electronic POP advertising 160 describedabove. For example, the communication unit 225 supplies data, such as anpicked-up image obtained by the image pickup unit 221, a distancemeasurement result obtained by the ranging unit 222, and the like asinformation associated with a customer and a product to the local server123. Furthermore, the communication unit 225 acquires data of an imageand voice relating to a product, for example, an advertisement of aproduct and the like, from the local server 123, supplies the image datato the display unit 223 and supplies the voice data to the voice outputunit 224. Note that the electronic POP advertising 160 may have astorage unit 226 that stores electric power, so that each processingunit is driven by using power of the storage unit 226. As a matter ofcourse, the configuration may be such that the storage unit 226 isomitted, and each processing unit is driven by using power supplied froman external power supply.

<Information to be Collected>

Information to be collected in the information collection system 100 isoptional. For example, information as shown in FIG. 10 may be collected.In a case of the example of FIG. 10, information of a time at which acustomer looks at a product, a time period in which a customer looks ata product, a distance between a customer and a product (which may be theinformation collection devices 122 or the product display shelf 121),attributes of a customer, such as gender, age, and the like, whether ornot a customer picks up a product, whether or not a customer returns apicked-up product to the product display shelf 121, and the like iscollected for each product.

As a matter of course, the configuration is not limited to the aboveexample, and any information may be collected. However, for example,information used for analysis, a sales promotion activity, and the likeas marketing information is preferably collected. With the informationcollection system 100 collecting more useful information, not onlyusability of information collection can be improved, but also an effectobtained by applying the present technology can be made larger.

<Information to be Transmitted>

The information collection system 100 generates information asexemplified in FIG. 10 by analyzing information (for example, apicked-up image, and the like) collected by the information collectiondevice 122. This analysis processing can be performed by an optionaldevice in the information collection system 100.

For example, in a case where the information collection system 100 hasthe configuration shown in FIG. 8 (described again in A of FIG. 11), theanalysis processing may be performed by the local server 123 (firstmode), the relay device 102 (the second mode), or the informationmanagement server 103 (the third mode) as specified in a table shown inB of FIG. 11. Like the table shown in B of FIG. 11, in a case of thefirst mode, data of a picked-up image is transmitted between theinformation collection devices 122 and the local server 123, and data ofa result of image analysis is transmitted between the local server 123and the relay device 102, and between the relay device 102 and theinformation management server 103. Furthermore, in a case of the secondmode, data of a picked-up image is transmitted between the informationcollection devices 122 and the local server 123, and between the localserver 123 and the relay device 102, and data of a result of imageanalysis is transmitted between the relay device 102 and the informationmanagement server 103. Moreover, in a case of the third mode, data of apicked-up image is transmitted between the information collectiondevices 122 and the local server 123, between the local server 123 andthe relay device 102, and between the relay device 102 and theinformation management server 103.

Note that the image analysis described above may be performed by theinformation collection device 122. In such a case, data of a result ofthe image analysis is transmitted between the information collectiondevice 122 and the local server 123, between the local server 123 andthe relay device 102, and between the relay device 102 and theinformation management server 103.

Furthermore, a frame rate at which image data is transmitted(transmission frame rate) is optional. For example, the frame rate maybe around 30 frames per second (fps), or around several fps (1 or 2fps). Furthermore, the transmission frame rate may be set as appropriatein accordance with the number of image sensors or the number of pixelsso that an image can be transmitted appropriately between devices.

Note that the configuration of the information collection system 100 isoptional, and is not limited to the example of FIG. 8. For example, asshown in A of FIG. 12, the configuration may be such that the localserver 123 of each of the shelf devices 101 is omitted, and theinformation collection device 122 and the relay device 102 perform wiredcommunication or radio communication, or both.

In this case, the analysis processing described above may be performedby the relay device 102 (first mode) or the information managementserver 103 (the second mode) as in a table shown in B of FIG. 12. Likethe table shown in B of FIG. 12, in a case of the first mode, data of apicked-up image is transmitted between the information collection device122 and the relay device 102, and data of a result of the image analysisis transmitted between the relay device 102 and the informationmanagement server 103. Furthermore, in a case of the second mode, dataof a picked-up image is transmitted either between the informationcollection device 122 and the relay device 102 and between the relaydevice 102 and the information management server 103.

Note that, in this case as well, the image analysis described above maybe performed by the information collection device 122. In such a case,data of a result of the image analysis is transmitted either between theinformation collection device 122 and the relay device 102 and betweenthe relay device 102 and the information management server 103.

Furthermore, in this case as well, a transmission frame rate of imagedata is optional. The transmission frame rate may be set as appropriatein accordance with the number of image sensors and the number of pixels.

Furthermore, for example, as shown in A of FIG. 13, the configurationmay be such that the relay device 102 is omitted, and the local server123 and the information management server 103 of each of the shelfdevices 101 perform wired communication or radio communication, or both.

In this case, the analysis processing described above may be performedby the local server 123 (first mode) or the information managementserver 103 (the second mode) as in a table shown in B of FIG. 13. Likethe table shown in B of FIG. 13, in a case of the first mode, data of apicked-up image is transmitted between the information collection device122 and the local server 123, and data of a result of the image analysisis transmitted between the local server 123 and the informationmanagement server 103. Furthermore, in a case of the second mode, dataof a picked-up image is transmitted either between the informationcollection device 122 and the local server 123 and between the localserver 123 and the information management server 103.

Note that, in this case as well, the image analysis described above maybe performed by the information collection device 122. In such a case,data of a result of the image analysis is transmitted either between theinformation collection device 122 and the local server 123 and betweenthe local server 123 and the information management server 103.

Furthermore, in this case as well, a transmission frame rate of imagedata is optional. The transmission frame rate may be set as appropriatein accordance with the number of image sensors and the number of pixels.

Furthermore, for example, as shown in A of FIG. 14, the configurationmay be such that the local server 123 and the relay device 102 areomitted, and the information collection device 122 of each of the shelfdevices 101 and the information management server 103 perform wiredcommunication or radio communication, or both.

In this case, the analysis processing described above may be performedby the information management server 103 (first mode) as in a tableshown in B of FIG. 14. Like the table shown in B of FIG. 14, in a caseof the first mode, data of a picked-up image is transmitted between theinformation collection device 122 and the information management server103.

Note that, in this case as well, the image analysis described above maybe performed by the information collection device 122. In such a case,data of a result of the image analysis is transmitted between theinformation collection device 122 and the information management server103.

Furthermore, in this case as well, a transmission frame rate of imagedata is optional. The transmission frame rate may be set as appropriatein accordance with the number of image sensors and the number of pixels.

<Power Supply>

Next, power supply to the information collection device 122 will bedescribed. The power supply to the information collection device 122 maybe performed from an external power supply. In such a case, for example,a configuration as shown in FIG. 15 may be employed. In a case of theexample of FIG. 15, a plug 251 is connected to an external AC powersupply. Furthermore, a lighting busway (JIS C8366) 252 is arranged in ahorizontal direction along a front surface of each shelf of the productdisplay shelf 121. Furthermore, an AC/DC converter 253 that converts anAC power supply to a DC power supply is provided between the lightingbusway 252 and each of the information collection devices 122. Then, anexternal AC power supply is supplied via the plug 251 and the lightingbusway 252, converted to a DC power supply by the AC/DC converter 253,and the DC power supply is supplied to each of the informationcollection devices 122.

Furthermore, a configuration as shown in FIG. 16 may be employed. In acase of the example of FIG. 16, a plug 261 is connected to an externalAC power supply. Furthermore, the plug 261 is connected to a radiofeeding cable 263 via an AC/DC converter 262. The radio feeding cable263 is installed along each shelf of the product display shelf 121 to beconnected to a base unit 264 of a connector for radio power feeding thatis arranged in each location on a front surface of each shelf.Furthermore, a remote unit (not shown) of a radio feeding connector isinstalled in each of the information collection devices 122. Then, theexternal AC power supply is supplied to the AC/DC converter 262 via theplug 261, and converted to a DC power supply. This DC power supply issupplied to the base unit 264 of a connector for radio power feeding viathe radio feeding cable 263. Then, the DC power supply is supplied to aremote unit of a radio feeding connector by radio power feeding from thebase unit 264 of the connector for radio power feeding, and is furthersupplied to the information collection device 122.

Furthermore, a configuration as shown in FIG. 17 may be employed. In acase of the example of FIG. 17, a plug 271 is connected to an externalAC power supply. Furthermore, the plug 271 is connected to a DC powerfeeding cable 273 via an AC/DC converter 272. The DC power feeding cable273 is installed along each shelf of the product display shelf 121 to beconnected to each of the information collection devices 122. Then, theexternal AC power supply is supplied to the AC/DC converter 272 via theplug 271, and converted to a DC power supply. This DC power supply issupplied to each of the information collection devices 122 via the DCpower feeding cable 273.

Furthermore, a configuration as shown in FIG. 18 may be employed. In acase of the example of FIG. 18, a plug 281 is connected to an externalAC power supply. Furthermore, the plug 281 is connected to the localserver 123 via an AC/DC converter 282. Furthermore, a power overEthernet cable 283 connected to the local server 123 is installed alongeach shelf of the product display shelf 121 connected to each of theinformation collection devices 122. Then, the external AC power supplyis supplied to the AC/DC converter 282 via the plug 281, converted to aDC power supply, and supplied to the local server 123. Furthermore, thisDC power supply is supplied to each of the information collectiondevices 122 from the local server 123 via the power over Ethernet cable283.

By employing the above configuration, power from an external powersupply can be supplied to the information collection device 122. In thismanner, larger power can be supplied to the information collectiondevice 122. Therefore, for example, in a case where the informationcollection device 122 has a monitor like the case of the electronicshelf label 150 and the electronic POP advertising 160, the monitor canbe caused to display advertisement information and the like morebrightly. Furthermore, for example, in a case where the informationcollection device 122 has a speaker like the case of an electronic POPadvertising, the speaker can be caused to output an advertising voicemore loudly. Accordingly, an effect as an advertisement can be obtainedmore significantly.

Note that the configuration may be such that the information collectiondevice 122 includes a storage battery as described above, and power isfed from the storage battery. By providing the storage battery, data canbe held (prevented from being erased) even in a case where theinformation collection device 122 is separated from a power supplysource due to, for example, movement of an arranged location, or thelike. Furthermore, the information collection device 122 in a state ofbeing separated from a power supply source can be driven, and, forexample, caused to collect information associated with a product and thelike (for example, pick up an image and the like).

<First Method of Identifying a Product Looked at by a Customer>

As described above, (an optional device of) the information collectionsystem 100 analyzes a picked-up image picked up by the informationcollection device 122 and identifies which product is looked at by acustomer. At this time, the information collection system 100 evaluatesor analyzes and identifies a direction of a face of a customer, so as toidentify which product is looked at by the customer. A specific exampleof this method of identifying a direction of a face will be describedhereinafter.

A of FIG. 19 is a diagram showing an embodiment in which informationcollection devices as many as product types arranged on the productdisplay shelf 121 on which products are displayed are arranged for eachproduct type arranged on the product display shelf 121. As an example ofthe embodiment, A of FIG. 19 is a diagram showing a state in which fivetypes of products (in other words, five product types, 131-1 to 131-5)are arranged on the product display shelf 121, information collectiondevices (122-1 to 122-5) as many as the number of (five) product typesare arranged for each product type, and one customer (301) stands infront of the product display shelf 121 and looks at the products whenviewed from above the shelf.

B of FIG. 19 shows an example of a picked-up image obtained by each ofthe information collection devices 122-1 to 122-5 in this case. In B ofFIG. 19, face images 302-1 to 302-5 are images of a face (head) portionof a customer 301 that are cut out from picked-up images obtained byeach of the information collection devices 122-1 to 122-5. In this case,only the customer 301 exists around the product display shelf 121, andall of these face images are face images of the customer 301.

The information collection system 100 compares a direction of a face ineach of the above face images. This comparison method is optional. Forexample, the configuration may be such that, as shown in C of FIG. 19, amark showing a positional relationship between a center of a face orhead area and a center of the face (for example, A1/W in C of FIG. 19)is calculated for each of the face images 302-1 to 302-5, and comparedbetween each of the images. Furthermore, for example, the configurationmay be such that, as shown in D of FIG. 19, a mark showing a positionalrelationship between both ends of a face or head and a center of theface (|B1−B2| or B1/B2 in D of FIG. 19) is calculated for each of theface images 302-1 to 302-5, and compared between each of the images.Moreover, for example, the configuration may be such that, as shown in Eof FIG. 19, a mark showing a positional relationship between a center ofa face and a center of each of two eyes (|C1−C2| or C1/C2 in E of FIG.19) is calculated for each of the face images 302-1 to 302-5, andcompared between each of the images.

Then, the information collection system 100 identifies an image of aface that is determined to “face the front the most” on the basis of aresult of the comparison. This identification method is optional. Forexample, in the case of C of FIG. 19, an image having a value of A1/Wclosest to zero may be identified (in other words, provided with aconclusion of evaluation or analysis) as an “image of a face facing thefront the most”. Furthermore, for example, in the case of D of FIG. 19,an image having a value of |B1−B2| closest to zero may be identified (inother words, provided with a conclusion of evaluation or analysis) as an“image of a face facing the front the most”, or an image having a valueof B1/B2 closest to one (image having a value of |B1/B2−1| closest tozero) may be identified (in other words, provided with a conclusion ofevaluation or analysis) as an “image of a face facing the front themost”. Moreover, for example, in the case of E of FIG. 19, an imagehaving a value of |C1−C2| closest to zero may be identified (in otherwords, provided with a conclusion of evaluation or analysis) as an“image of a face facing the front the most”, or an image having a valueof C1/C2 closest to one (image having a value of |C1/C2−1| closest tozero) may be identified (in other words, provided with a conclusion ofevaluation or analysis) as an “image of a face facing the front themost”.

Then, the information collection system 100 identifies the product 131corresponding to the information collection device 122 that picks up animage that is identified as “facing the front the most” as a “productlooked at by a customer”.

For example, in the case of B of FIG. 19, the face image 302-3 isidentified as an “image of a face facing the front the most”. Since theface image 302-3 is an image cut out from a picked-up image picked up bythe information collection devices 122-3, the customer 301 is identifiedas looking at the product 131-3 corresponding to the informationcollection devices 122-3.

A configuration that, by comparing a plurality of images picked up by aplurality of the information collection devices 122, an image of a faceof a customer facing the front the most is identified, and, by doing so,a product looked at by the customer is identified, will be describedmore in detail.

First, as a first case, a case where a customer looks (looks straight,looks directly) at the product type 131 other than those on both ends onthe product display shelf 121 will be described. The informationcollection device 122 is arranged in the vicinity of each of a pluralityof the product types 131 arranged on the product display shelf 121. Eachof the information collection devices 122 is associated (linked) witheach of the product types 131 in the vicinity. An image of a face of acustomer looking at a certain product type arranged on the productdisplay shelf 121 is picked up by a plurality of the informationcollection devices.

Then, by using A1/W, |B1−B21, |C1−C2|, B1/B2−1|, or |C1/C2−1| describedabove as a mark, images picked up by the information collection device122 are compared (more specifically, values of marks of the images arecompared) in the order of arrangement of each of the informationcollection devices 122 on the product display shelf 121. In the firstcase, the above mark has a certain positive value in an image picked upby the information collection device 122 associated with the producttype 131 arranged on one end of the display shelf. After the above, ifvalues of the marks in images picked up by the information collectiondevice 122 are compared in the order of arrangement of each of theinformation collection devices 122 on the product display shelf 121,values of the marks decrease monotonously as it is closer to a producttype looked at by the customer, and a value of the mark becomes asmallest value (a minimum value, or a value significantly close to zero,with some variations depending on an attitude of the customer looking ata product). Thereafter, as it is farther from the product type looked atby the customer, values of the marks monotonously increase.

For this reason, if images picked up by the information collectiondevice 122 are compared in the order of arrangement of each of theinformation collection devices 122 on the product display shelf 121 byusing A1/W, |B1−B2|, |C1−C2|, B1/B2−1|, or |C1/C2−1| described above asa mark, the product type 131 associated (linked) with the informationcollection devices 122 that picks up an image having a value of theabove mark at a smallest value (a minimum value, or a valuesignificantly close to zero, with some variations depending on anattitude of the customer looking at a product) can be identified as aproduct that is looked (looked straight, looked directly) at by thecustomer. At the same time, in an image picked up by the informationcollection device 122 associated (linked) with the product type 131identified above among all product types arranged on a display shelf,that the mark described above relating to a direction of a face of thecustomer is smallest is confirmed in an image obtained by picking up aface of the customer. Accordingly, identification of a product looked(looked straight, looked directly) at by the customer is confirmed andverified to be correct.

Next, as a second case, a case where a customer looks (looks straight,looks directly) at either one of the product types 131 at both ends onthe product display shelf 121 will be described. In this case, in asimilar manner as the first case, values of the above mark (A1/W,|B1−B2|, |C1−C2|, B1/B2−1|, or |C1/C2−1|) in images picked up by theinformation collection devices 122 are compared in the order ofarrangement of the information collection devices 122 on the productdisplay shelf 121.

As a result, (1) a value of the mark described above becomes a smallestvalue (a minimum value, or a value significantly close to zero, withsome variations depending on an attitude of the customer looking at aproduct) in an image picked up by the information collection device 122associated with the product type 131 arranged on one end (first end) ona display shelf, and, thereafter, as a distance from the first end tothe information collection device 122 becomes larger, a value of themark monotonously increases, or (2) a value of the mark described abovebecomes a certain positive value in an image picked up by theinformation collection device 122 associated with the product type 131arranged on one end (first end) on a display shelf, and, thereafter, avalue of the mark monotonously decreases as a distance from the firstend to the information collection device 122 becomes larger, and a valueof the mark described above becomes a smallest value (a minimum value,or a value significantly close to zero, with some variations dependingon an attitude of the customer looking at a product) in an image pickedup by the information collection device 122 associated with the producttype 131 arranged on the other end (second end) on the display shelf.

For this reason, if images picked up by the information collectiondevice 122 are compared in the order of arrangement of each of theinformation collection devices 122 on the product display shelf 121 byusing A1/W, |B1−B2|, |C1−C2|, B1/B2−1|, or |C1/C2−1| described above asa mark, the product type 131 associated (linked) with the informationcollection devices 122 that picks up an image having a value of theabove mark at a smallest value (a minimum value, or a valuesignificantly close to zero, with some variations depending on anattitude of the customer looking at a product) can be identified as aproduct that is looked (looked straight, looked directly) at by thecustomer. At the same time, in an image picked up by the informationcollection device 122 associated (linked) with the product type 131identified above among all product types arranged on a display shelf,that the mark described above relating to a direction of a face of thecustomer is smallest is confirmed in an image obtained by picking up aface of the customer. Accordingly, identification of a product looked(looked straight, looked directly) at by the customer is confirmed andverified to be correct.

Moreover, the information collection system 100 measures a time periodin which the customer 301 looks at the product. Furthermore, theinformation collection system 100 acquires attributes, such as age,gender, and the like, of the customer 301 who looks at the product byusing an optional analysis method. Moreover, the information collectionsystem 100 finds a distance between the customer 301 and a productlooked at by the customer 301 (which may be the information collectiondevice 122 or the product display shelf 121 corresponding to theproduct). The information collection system 100 collects these analysisresults as information associated with a customer.

Even in a case where the same person is included in a plurality ofpicked-up images, the number of directions (products) looked at by onecustomer is only one. Accordingly, the number of picked-up images inwhich a face image is determined to face the front the most is one.Accordingly, by identifying a product looked at by a customer(identifying a direction of a face of the customer) in the above manner,a face image of the same person does not need to be identified from eachpicked-up image, and a product looked at by the customer can beidentified more easily and with higher accuracy. Furthermore, by using adirection of a face, a product looked at by a customer can be identifiedwith a simpler algorithm and with higher accuracy than in a case where adestination of a gaze of a customer is estimated. That is, theinformation collection system 100 can collect information with higheraccuracy more easily. Moreover, that a customer faces the front (thatthe mark with respect to a direction of a face described above issmallest) is confirmed in an image obtained by picking up a face of thecustomer. Accordingly, that the above identification of a product lookedat by the customer is correct is confirmed and verified.

Note that the configuration may be such that a product looked at by acustomer is estimated by estimating a direction of a gaze of a customer.In this case as well, since the information collection system 100includes the information collection device 122 for each product type,images can be picked up from more directions than a conventionaltechnique with respect to a customer. For this reason, information withhigher accuracy can be collected.

<Second Method of Identifying a Product Looked at by a Customer>

Note that the configuration may be such that “whether or not a face isfacing this direction (whether or not a direction of a face faces thefront)” is determined by identifying a direction of the face in eachpicked-up image in place of comparing a direction of a face betweenpicked-up images.

This determination method is optional. For example, the configurationmay be such that the information collection system 100 calculates a mark(A1/W in C of FIG. 19) showing a positional relationship between acenter of a face or head region and a center of the face, and makesdetermination on the basis of a value of the mark. Furthermore, forexample, the configuration may be such that the information collectionsystem 100 calculates a mark (B1/B2 in D of FIG. 19) showing apositional relationship between both ends of a face or head region and acenter of the face, and makes determination on a basis of the value ofthe mark. Moreover, for example, the configuration may be such that theinformation collection system 100 calculates a mark (C1/C2 in E of FIG.19) showing a positional relationship between a center of a face and acenter of each of two eyes, and makes determination on the basis of avalue of the mark.

A determination condition of each of the determination methods isoptional. For example, in the case of C of FIG. 19, if the value of A1/Wis close to zero, an image can be determined as an “image of a facefacing the front”. Accordingly, a face may be determined to be “facingthe front” in a case where the value of A1/W is within a range of “0±α”.Furthermore, in the case of D of FIG. 19, if the value of B1/B2 is closeto one, an image can be determined as an “image of a face facing thefront”. Accordingly, a face may be determined to be “facing the front”in a case where the value of B1/B2 is within a range of “1±β”. Moreover,in the case of E of FIG. 19, if the value of C1/C2 is close to one, animage can be determined as an “image of a face facing the front”.Accordingly, a face may be determined to be “facing the front” in a casewhere the value is within a range of “1±γ”. Note that the above α, β,and γ, which depend on space between installed information collectiondevices (or products) and accuracy of detection and analysis of a face,may be adjustable in accordance with such parameters.

For example, in the case of B of FIG. 19, only the face image 302-3 isdetermined as an “image of a face facing the front”. Since the faceimage 302-3 is an image cut out from a picked-up image picked up by theinformation collection devices 122-3, the customer 301 is identified aslooking at the product 131-3 corresponding to the information collectiondevices 122-3.

Moreover, the information collection system 100 measures a time periodin which the customer 301 looks at the product. Furthermore, theinformation collection system 100 acquires attributes, such as age,gender, and the like, of the customer 301 who looks at the product byusing an optional analysis method. Moreover, the information collectionsystem 100 finds a distance between the customer 301 and a productlooked at by the customer 301 (which may be the information collectiondevice 122 or the product display shelf 121 corresponding to theproduct). The information collection system 100 collects these analysisresults as information associated with a customer.

Even in a case where the same person is included in a plurality ofpicked-up images, the number of directions (products) looked at by onecustomer is only one. Accordingly, the number of picked-up images inwhich a face image is determined to face the front is one. Accordingly,by identifying a product looked at by a customer (identifying adirection of a face of the customer) in the above manner, a face imageof the same person does not need to be identified from each picked-upimage, and a product looked at by the customer can be identified moreeasily and with higher accuracy. Furthermore, by using a direction of aface, a product looked at by a customer can be identified with a simpleralgorithm and with higher accuracy than in a case where a destination ofa gaze of a customer is estimated. That is, the information collectionsystem 100 can collect information with higher accuracy more easily.

Note that the configuration may be such that a product looked at by acustomer is estimated by estimating a direction of a gaze of a customer.In this case as well, since the information collection system 100includes the information collection device 122 for each product type,images can be picked up from more directions than a conventionaltechnique with respect to a customer. For this reason, information withhigher accuracy can be collected.

<First Identification Method in a Case of a Plurality of Customers>

Next, a case of a plurality of customers will be described. In a case ofan example shown in A of FIG. 20, three customers 311 (a customer 311A,a customer 311B, and a customer 311C) stand around the product displayshelf 121.

In this case, a face image of each of the customers is included in anpicked-up image picked up by each of the information collection devices122 as shown in an example of B of FIG. 20. For example, a picked-upimage obtained by the information collection device 122-1 includes aface image 312-1 of the customer 311A and a face image 313-1 of thecustomer 311B. Furthermore, a picked-up image obtained by theinformation collection device 122-2 includes a face image 312-2 of thecustomer 311A and a face image 313-2 of the customer 311B. Moreover, apicked-up image obtained by the information collection device 122-3includes a face image 312-3 of the customer 311A, a face image 313-3 ofthe customer 311B, and a face image 314-1 of the customer 311C.Furthermore, a picked-up image obtained by the information collectiondevice 122-4 includes a face image 312-4 of the customer 311A, a faceimage 313-4 of the customer 311B, and a face image 314-2 of the customer311C. Moreover, a picked-up image obtained by the information collectiondevices 122-5 includes a face image 312-5 of the customer 311A, a faceimage 313-5 of the customer 311B, and a face image 314-3 of the customer311C.

That is, a plurality of face images of the same person are included inthe above picked-up images. In view of the above, the informationcollection system 100 identifies the “same person” picked up in aduplicate manner in face images picked up by each of the informationcollection devices 122, and creates a group by gathering the “sameperson” (a dotted line frame in B of FIG. 20).

Then, the information collection system 100 compares directions of facesbetween each of the face images that are grouped as the “same person”.This method of comparing directions of faces is optional. For example,comparison may be made by the method described with reference to FIG.19. In a case of the example of FIG. 20, the customer 311A is identifiedas looking at the information collection device 122-3 (that is, theproduct 131-3) since the face image 312-3 “faces the front the most”.Furthermore, the customer 311B is identified as looking at theinformation collection devices 122-2 (that is, the product 131-2) sincethe face image 313-2 “faces the front the most”. Moreover, the customer311C is identified as looking at the information collection devices122-5 (that is, the product 131-5) since the face image 314-3 “faces thefront the most”.

In a similar manner as the case of FIG. 19, the information collectionsystem 100 measures a time period in which each of the customers 311A to311C looks at a product. Furthermore, the information collection system100 acquires attributes, such as age, gender, and the like, of thecustomers 311A to 311C who look at a product by using an optionalanalysis method. Moreover, the information collection system 100 finds adistance between the customers 311A to 311C and a product looked at byeach of the customers (which may be the information collection device122 or the product display shelf 121 corresponding to the product). Theinformation collection system 100 collects these analysis results asinformation associated with a customer.

By identifying a product looked at by a customer (identifying adirection of a face of a customer) in the above manner, the productlooked at by the customer can be identified by a simpler algorithm andwith higher accuracy than a case where a destination of a gaze of acustomer is estimated, even in a case where a plurality of customersexist around the product display shelf 121. That is, the informationcollection system 100 can collect information with higher accuracy moreeasily.

Note that the configuration may be such that a product looked at by acustomer is estimated by estimating a direction of a gaze of a customer.In this case as well, since the information collection system 100includes the information collection device 122 for each product type,images can be picked up from more directions than a conventionaltechnique with respect to a customer. For this reason, information withhigher accuracy can be collected.

<First Identification of the Same Person>

Next, an example of a method of identifying the same person from animage of a customer in the method described above will be described. Themethod of identifying the same person is optional. For example,identification of the same person may be performed by using a rangingsensor of the information collection device 122.

For example, as shown in A of FIG. 21, assume that the products 131-1 to131-5 are displayed on the product display shelf 121, and theinformation collection devices 122-1 to 122-5 are installed in a mannercorresponding to the products. Then, assume that customers A to G standaround the front of the product display shelf 121. Here, for the purposeof description, an X-coordinate and a Y-coordinate are set with respectto front space of the product display shelf 121 as shown in A of FIG.21. Then, coordinates of a nose of the customer A are set to (x1, y1),coordinates of a nose of the customer B are set to (x2, y2) coordinatesof a nose of the customer C are set to (x3, y3), coordinates of a noseof the customer D are set to (x4, y4), coordinates of a nose of thecustomer E are set to (x5, y5), coordinates of a nose of the customer Fare set to (x6, y6), and coordinates of a nose of the customer G are setto (x7, y7).

Each of the information collection devices 122 measures an angle and adistance for a face (or a head) detected in each picked-up image byusing a ranging sensor mounted on each of the information collectiondevices 122 as shown in B of FIG. 21. Specifically, an angle and adistance to an object (a face, for example, a position of a nose) aremeasured. The information collection system 100 finds the X and Ycoordinates on the basis of a result of the measurement. Sincecoordinates of the same person are the same, the same person can beidentified on the basis of the coordinates.

For example, as shown in C of FIG. 21, all face images for whichcoordinates of (x5, y5) are found can be identified as face images ofthe customer E. Furthermore, all face images for which coordinates of(x6, y6) are found can be identified as face images of the customer F.Moreover, all face images for which coordinates of (x7, y7) are foundcan be identified as face images of the customer G. The informationcollection system 100 creates a group of the “same face” with respect toall detected faces, and uses the group at the time of comparing facedirections.

In the above manner, the information collection system 100 can identifythe same person easily and with high accuracy.

<Second Identification of the Same Person>

Furthermore, the configuration may be such that the same person isidentified by performing monitoring for a certain period of time (frompicked-up images of a plurality of frames).

For example, as shown in A of FIG. 22, assume that the products 131-1 to131-5 are displayed on the product display shelf 121, and theinformation collection devices 122-1 to 122-5 are installed in a mannercorresponding to the products. Then, assume that customers 331A and 331Bstand around the front of the product display shelf 121.

At a certain timing, picked-up images obtained by each of theinformation collection devices 122 are as shown in B of FIG. 22. Thatis, in a picked-up image 332-1 obtained by the information collectiondevice 122-1, face images of the customers 331A and 331B are notdetected. Furthermore, in a picked-up image 332-2 obtained by theinformation collection device 122-2, a face image 333-1 of the customer331A, and a face image 334-1 of the customer 331B are detected.Moreover, in a picked-up image 332-3 obtained by the informationcollection device 122-3, a face image 333-2 of the customer 331A, and aface image 334-2 of the customer 331B are detected. Furthermore, in apicked-up image 332-4 obtained by the information collection devices122-4, a face image 333-3 of the customer 331A and a face image 334-3 ofthe customer 331B are detected. Furthermore, in a picked-up image 332-5obtained by the information collection device 122-5, a face image 333-4of the customer 331A is detected.

From here, a focus will be placed on movement of a customer in a timedirection of each picked-up image in order to identify the “sameperson”. For the “same person”, existence or non-existence of movementand directions of movement at the same time are in the same tendencyeven in images between different cameras. The information collectionsystem 100 identifies the same person by using this tendency.

For example, in a case of FIG. 23, in each picked-up image, three facesof the customer A enclosed by a solid line frame start to move togetherat a time Ti−1, stop together at a time Ti, and, thereafter, all of thethree faces are stationary until a time Ti1+1. That is, these three faceimages move in one set as indicated by a solid arrow. The informationcollection system 100 identifies a person in the same series ofmovements, particularly when movement is started, when a moving actionis stopped, and moving directions, as the “same person” as describedabove.

Furthermore, in each picked-up image, three faces of the customer Benclosed by a dotted line frame are stopped until the time Ti, start tomove together at the time Ti, and have moved together to illustratedlocations at the time Ti+1. That is, these three face images move in oneset as indicated by a dotted arrow. The information collection system100 identifies a person in the same series of movements, particularlywhen movement is started, when a moving action is stopped, and movingdirections, as the “same person” as described above.

Even if a plurality of people move at the same timing, locations inwhich a plurality of these different people walk are different from eachother. Therefore, timings at which a plurality of these people move intoa frame of each camera are different from each other. Therefore, whichfaces picked up by each camera are of the same person can be determined.

The information collection system 100 creates a group of the “same face”with respect to all detected faces, and uses the group at the time ofcomparing face directions.

In the above manner, the information collection system 100 can identifythe same person easily and with high accuracy.

Note that, the configuration may be such that the information collectionsystem 100 identifies the same person by other optional methods, forexample, a method of identifying which faces are of the “same person” byface recognition, or the like.

<Second Identification Method in a Case of a Plurality of Customers>

Note that the configuration may be such that “whether or not a face isfacing this direction (whether or not a direction of a face faces thefront)” is determined by identifying a direction of the face in eachpicked-up image as described in <Second method of identifying a productlooked at by a customer> even in a case where a plurality of customersare included in a picked-up image. In this case, “whether or not a faceis facing this direction (whether or not a direction of a face faces thefront)” is preferably determined by identifying a direction of the facefor each face image in each picked-up image.

For example, as shown in A of FIG. 24, assume that the products 131-1 to131-5 are displayed on the product display shelf 121, and theinformation collection devices 122-1 to 122-5 are installed in a mannercorresponding to the products. Then, assume that customers 361A, 361B,and 361C stand around the front of the product display shelf 121.

At a certain timing, picked-up images obtained by each of theinformation collection devices 122 are as shown in B of FIG. 24. Thatis, in a picked-up image 362-1 obtained by the information collectiondevice 122-1, a face image 363-1 of the customer 361A and a face image364-1 of the customer 361B are detected. Furthermore, in a picked-upimage 362-2 obtained by the information collection devices 122-2, a faceimage 363-2 of the customer 361A, a face image 364-2 of the customer361B, and a face image 365-1 of the customer 361C are detected.Moreover, in a picked-up image 362-3 obtained by the informationcollection devices 122-3, a face image 363-3 of the customer 361A, aface image 364-3 of the customer 361B, and a face image 365-2 of thecustomer 361C are detected. Furthermore, in a picked-up image 362-4obtained by the information collection devices 122-4, a face image 363-4of the customer 361A, a face image 364-4 of the customer 361B, and aface image 365-3 of the customer 361C are detected. Moreover, in apicked-up image 362-5 obtained by the information collection device122-5, a face image 363-5 of the customer 361A, a face image 364-5 ofthe customer 361B, and a face image 365-4 of the customer 361C aredetected.

The information collection system 100 determines “whether or not a faceis facing this direction” (specifically, “whether or not a direction ofa face faces the front”) by identifying directions of all of thedetected faces in each of the picked-up images 362. This determinationmethod is optional. For example, determination may be made by a methodsimilar to the method described in <Second method of identifying aproduct looked at by a customer>.

For example, in a case of B of FIG. 24, the information collectionsystem 100 determines that a “face facing the front” “does not exist”for the picked-up image 362-1. Furthermore, the information collectionsystem 100 determines that a “face facing the front” “exists” (the faceimage 363-2) for the picked-up image 362-2, and determines that “thecustomer 361A looks at the product 131-2”. Moreover, the informationcollection system 100 determines that a “face facing the front” “exists”(the face image 364-3) for the picked-up image 362-3, and determinesthat “the customer 361B looks at the product 131-3”. Furthermore, theinformation collection system 100 determines that a “face facing thefront” “does not exist” for the picked-up image 362-4. Moreover, theinformation collection system 100 determines that a “face facing thefront” “exists” (the face image 365-4) for the picked-up image 362-5,and determines that “the customer 361C looks at the product 131-5”.

Moreover, the information collection system 100 measures a time periodin which the customer 361 looks at the product. Furthermore, theinformation collection system 100 acquires attributes, such as age,gender, and the like, of the customer 361 who looks at the product byusing an optional analysis method. Moreover, the information collectionsystem 100 finds a distance between the customer 361 and a productlooked at by the customer 361 (which may be the information collectiondevice 122 or the product display shelf 121 corresponding to theproduct). The information collection system 100 collects these analysisresults as information associated with a customer.

By identifying a product looked at by a customer (identifying adirection of a face of a customer) in the above manner, the productlooked at by the customer can be identified more easily and with highaccuracy, even in a case where a plurality of customers exist around theproduct display shelf 121. Furthermore, by using a direction of a face,a product looked at by a customer can be identified with a simpleralgorithm and with higher accuracy than in a case where a destination ofa gaze of a customer is estimated. That is, since the informationcollection system 100 includes the information collection device 122 foreach product type, images can be picked up from more directions than aconventional technique with respect to a customer. For this reason,information with higher accuracy can be collected more easily.

<First Identification Method in a Vertical Direction>

In the above description, identification of a direction of a face of acustomer in a horizontal direction is described. This identification maybe performed also in a vertical direction.

For example, as shown in A of FIG. 25, assume that the informationcollection devices 122-1 to 122-5 are installed on each shelf of theproduct display shelf 121. That is, assume that the informationcollection devices 122-1 to 122-5 are arranged in a vertical direction.Then, assume that a customer 371 is around the front of the productdisplay shelf 121.

Furthermore, assume that each face image shown in B of FIG. 25 isdetected from a picked-up image obtained by each of the informationcollection devices 122-1 to 122-5. That is, assume that a face image372-1 is detected in a picked-up image obtained by the informationcollection device 122-1, a face image 372-2 is detected in a picked-upimage obtained by the information collection device 122-2, a face image372-3 is detected in a picked-up image obtained by the informationcollection device 122-3, a face image 372-4 is detected in a picked-upimage obtained by the information collection device 122-4, and a faceimage 372-5 is detected in a picked-up image obtained by the informationcollection device 122-5.

The information collection system 100 compares directions of faces ineach of the above face images. This comparison method is optional. Forexample, the configuration may be such that the information collectionsystem 100 calculates a ratio of a length between an eye and an upperlip to a width of a face (N/W in a face image 372 in C of FIG. 25) foreach of the face images 372-1 to 372-5, and compares the mark (N/W)between the images.

Then, the information collection system 100 identifies an image of aface that is determined to “face the front the most” on the basis of aresult of the comparison. This identification method is optional. Forexample, in the case of C of FIG. 25, an image having a largest value ofN/W may be identified (in other words, provided with a conclusion ofevaluation or analysis) as an “image of a face facing the front themost”. According to this method, in a case of an example of B of FIG.25, the face image 372-4 in a picked-up image obtained by theinformation collection devices 122-4 is identified as an “image of aface facing the front the most”.

Then, the information collection system 100 identifies the product 131corresponding to the information collection device 122 that picks up animage that is identified as “facing the front the most” as a “productlooked at by a customer”.

Moreover, the information collection system 100 measures a time periodin which the customer 371 looks at the product. Furthermore, theinformation collection system 100 acquires attributes, such as age,gender, and the like, of the customer 371 who looks at the product byusing an optional analysis method. Moreover, the information collectionsystem 100 finds a distance between the customer 371 and a productlooked at by the customer 371 (which may be the information collectiondevice 122 or the product display shelf 121 corresponding to theproduct). The information collection system 100 collects these analysisresults as information associated with a customer.

In a case of a vertical direction as well, there is only one direction(product) looked at by one customer. Accordingly, in this case as well,by performing identification in the above manner, a face image of thesame person does not need to be identified from each picked-up image,and a product looked at by the customer can be identified more easilyand with higher accuracy. Furthermore, a product looked at by a customercan be identified only by comparing directions of faces. That is, aproduct looked at by a customer can be identified with a simpleralgorithm and with higher accuracy than in a case where a destination ofa gaze of a customer is estimated. That is, the information collectionsystem 100 can collect information with higher accuracy more easily.

Note that the configuration may be such that a product looked at by acustomer is estimated by estimating a direction of a gaze of a customer.In this case as well, since the information collection system 100includes the information collection device 122 for each product type,images can be picked up from more directions than a conventionaltechnique with respect to a customer. For this reason, information withhigher accuracy can be collected.

<Second Identification Method in a Vertical Direction>

The method of identifying a direction of a face of a customer in avertical direction is not limited to the example described above. Forexample, a direction of a face may be identified by evaluating a mark ineach image without comparing a mark between face images.

For example, in the case of A of FIG. 25, assume that the face images372-1 to 372-5 as shown in B of FIG. 25 are detected from each picked-upimage in a manner similar to the case described in <First identificationmethod in a vertical direction>. The information collection system 100identifies directions of these faces and determines “whether or not aface looks in this direction”. More specifically, the informationcollection system 100 determines “whether or not a direction of a facefaces the front”. In this case, comparison with face images in the otherpicked-up images is not performed.

Note that this method of identifying a direction of a face is optional.For example, as shown in C of FIG. 25, the configuration may be suchthat the information collection system 100 calculates a ratio of alength between an eye and an upper lip to a width of a face (N/W in C ofFIG. 25) for the face image 372, and makes determination on the basis ofthe value.

For example, assume that statistical average values of N and W are NOand W0, respectively. If N/W, which is a mark of a direction of a faceis close to N0/W0, an image can be determined as an “image of a facefacing the front”. Accordingly, in a case where the value is within arange of “N0/W0±α”, the image may be determined as a face “facing thefront”. Note that this a, which may be an optional value, may also be,for example, 10% of the value of NO/W0. Furthermore, a value of a, whichdepends on space between products or installed information collectiondevices 122 and accuracy of detection and analysis of a face, may beadjustable.

Moreover, the information collection system 100 measures a time periodin which the customer 371 looks at the product. Furthermore, theinformation collection system 100 acquires attributes, such as age,gender, and the like, of the customer 371 who looks at the product byusing an optional analysis method. Moreover, the information collectionsystem 100 finds a distance between the customer 371 and a productlooked at by the customer 371 (which may be the information collectiondevice 122 or the product display shelf 121 corresponding to theproduct). The information collection system 100 collects these analysisresults as information associated with a customer.

In this case as well, there is only one direction (product) looked at byone customer. Accordingly, in this case as well, by performingidentification in the above manner, a face image of the same person doesnot need to be identified from each picked-up image, and a productlooked at by the customer can be identified more easily and with higheraccuracy. Furthermore, by using a direction of a face, a product lookedat by a customer can be identified with a simpler algorithm and withhigher accuracy than in a case where a destination of a gaze of acustomer is estimated. That is, the information collection system 100can collect information with higher accuracy more easily.

Note that the configuration may be such that a product looked at by acustomer is estimated by estimating a direction of a gaze of a customer.In this case as well, since the information collection system 100includes the information collection device 122 for each product type,images can be picked up from more directions than a conventionaltechnique with respect to a customer. For this reason, information withhigher accuracy can be collected.

<Regarding Information Display>

For example, in a case where an information output (display) function isincluded like an electronic shelf label, an electronic POP advertising,and the like, the information collection device 122 can output (display,and the like) information. For example, this information may be storedin advance in a storage unit incorporated in the information collectiondevice 122, or may be input to the information collection device 122 bythe user and the like.

Furthermore, the information may be supplied from other devices via acommunication function. For example, as shown in FIG. 26, theinformation may be supplied from the information management server 103(the information collection device 122 may output information suppliedfrom the information management server 103). Note that communicationbetween the information collection device 122 and the informationmanagement server 103 does not need to be performed via the local server123, the relay device 102, or the like, and illustration and descriptionof these are omitted in FIG. 26. As described above, the informationcollection device 122 supplies collected information (informationassociated with a product and a customer, and the like) to theinformation management server 103 by using a communication function (anarrow 401). Then, the information management server 103 suppliesinformation to the information collection device 122 by using thiscommunication function (which, as a matter of course, may be othercommunication functions) (an arrow 402). The information collectiondevice 122 outputs the information supplied from the informationmanagement server 103 as, for example, an image, voice, and the like.

At that time, the information management server 103 may supplyinformation associated with information supplied from the informationcollection device 122 to the information collection device 122. Forexample, if the information collection device 122 picks up an image of aproduct, identifies the product on the basis of the picked-up image, andsupplies identification information of the product to the informationmanagement server 103, the information management server 103 suppliesinformation associated with a product corresponding to theidentification information (for example, a product name, a price, andthe like) (that is, information associated with a product whose imagehas been picked up) to the information collection device 122. Theinformation collection device 122 outputs the product name, price, andthe like as an image (including character information), voice, and thelike. Only by causing the information collection device 122 to pick upan image of a product, information of the product name, price, and thelike can be easily output to (displayed on, and the like) theinformation collection device 122.

Furthermore, for example, if the information collection device 122 picksup an image of a customer, identifies attributes of the customer on thebasis of the picked-up image, and supplies attribute information of thecustomer to the information management server 103, the informationmanagement server 103 supplies information associated with a productcorresponding to the attribute information (for example, a product name,a price, and the like recommended to a customer of the attributes) (thatis, information to be presented to the customer whose image has beenpicked up) to the information collection device 122. The informationcollection device 122 outputs the product name, price, and the like asan image (including character information), voice, and the like. In theabove manner, the information collection device 122 can easily output(display, and the like) information appropriate for a customerpositioned around.

Note that, in the example of FIG. 26, the information management server103 may supply information (independent information) not related toinformation supplied from the information collection device 122 to theinformation collection device 122. In this case, transmission times ofinformation may be independent of each other. That is, the informationmanagement server 103 may supply information independent of informationfrom the information collection device 122 to the information collectiondevice 122 at a timing independent of a supplying timing of theinformation from the information collection device 122.

For example, the information management server 103 supplies informationnot related to information supplied from the information collectiondevice 122, such as guide information on a selling place, guideinformation on a product on sale or scheduled to be on sale for alimited time, and the like, to the information collection device 122 ata timing not related to an information supply timing from theinformation collection device 122 (for example, even if information isnot supplied from the information collection device 122), and theinformation collection device 122 outputs the information as an image(including character information), voice, and the like. In this manner,optional information can be output to the information collection device122 more easily. As a matter of course, the information managementserver 103 can optionally select the information collection device 122to which such information is supplied. That is, optional information canbe easily output to an optional location (the information collectiondevice 122).

Note that information output by the information collection device 122may be supplied from a device other than the information managementserver 103. For example, as shown in FIG. 27, the information may besupplied from an information provision server 411 that is separate fromthe information management server 103.

That is, in this case, the information collection device 122 suppliescollected information (information associated with a product and acustomer, and the like) to the information management server 103 byusing a communication function (an arrow 421). Furthermore, theinformation provision server 411 supplies information to the informationcollection devices 122 by using this communication function (which, as amatter of course, may be other communication functions) (an arrow 422).The information collection device 122 outputs the information suppliedfrom the information provision server 411 as, for example, an image,voice, and the like.

In a similar manner as a case of the example of FIG. 26, informationsupplied from the information collection device 122 to the informationmanagement server 103 and information supplied from the informationprovision server 411 to the information collection device 122 may berelated to each other, or independent of each other. Furthermore,transmission timings of these pieces of information may also be relatedto each other or independent of each other.

Moreover, a plurality of the information provision servers 411 may beprovided. That is, the configuration may be such that the informationcollection device 122 can output information supplied from a pluralityof devices.

Furthermore, the information management server 103 in FIG. 26 and theinformation provision server 411 in FIG. 27 that supply information tobe output by the information collection device 122 may supply optionalinformation to an optional one of the information collection device 122at an optional timing. For example, the configuration may be such thatthe same information can also be supplied to a plurality of theinformation collection devices 122, or information related toinformation supplied from a certain one of the information collectiondevice 122 can be supplied to the information collection device 122, andinformation not related to the information supplied from the informationcollection device 122 can be supplied to the other informationcollection devices 122. Furthermore, the configuration may be such thatinformation is not supplied to part of the information collectiondevices 122. Furthermore, for example, the configuration may be suchthat information associated with information supplied from a certain oneof the information collection devices 122 to the other informationcollection devices 122.

Note that, in the examples of FIGS. 26 and 27, the configuration may besuch that the information collection device 122 does not have a functionof supplying information to the information management server 103.Information collected by the information collection device 122 may betaken out from the information collection device 122 by an optionalmethod. For example, the configuration may be such that collectedinformation is stored in a removable medium, such as a USB memory andthe like, and is output to the outside (for example, other devices, suchas the information management server 103 and the like) of theinformation collection devices 122 via the removable medium.

2. Second Embodiment

<Electronic Shelf Label System>

Next, a more specific example of the information processing systemdescribed above will be described. FIG. 28 is a diagram showing a mainconfiguration example of an electronic shelf label system that is anembodiment of an information processing system to which the presenttechnology is applied. An electronic shelf label system 1100 shown inFIG. 28 is a system that causes an electronic shelf label 1101 todisplay information of a product.

As shown in FIG. 28, the electronic shelf label system 1100 has N (N isan optional natural number) of the electronic shelf labels 1101(electronic shelf labels 1101-1 to 1101-N). In a case where theelectronic shelf labels 1101-1 to 1101-N are not distinguished from eachother, such electronic shelf labels are referred to as the electronicshelf labels 1101. As shown on the electronic shelf label 1101-1, ahousing of the electronic shelf label 1101 is provided with a camera1111, a monitor 1112, and a mode identification light emitting diode(LED) 1113.

The camera 1111 has an optional image pickup element, for example, acharge coupled device (CCD) image sensor, a complementary metal oxidesemiconductor (CMOS) image sensor, or the like, and picks up an image ofan object in front of the camera 1111. Note that the configuration maybe such that the camera 1111 has an optional optical system, forexample, a lens, a diaphragm, and the like, in addition to the imagepickup element.

The monitor 1112 has an optional display device, for example, a liquidcrystal display (LCD), an organic electro luminescence display (OELD),or the like, and displays optional information relating to a product,for example, a product name, a category, a price, identificationinformation (for example, an identification number, a barcode, and thelike), and the like.

The mode identification LED 1113 is a display device that shows anoperation mode of the electronic shelf label 1101. The modeidentification LED 1113 shows an operation mode of the electronic shelflabel 1101 by, for example, a luminescent color, existence ornon-existence of luminescence, and the like. Note that a display devicethat shows an operation mode of the electronic shelf label 1101 is notlimited to an LED, and may be configured with an optional displaydevice. Furthermore, for example, the configuration may be such that theabove display device is omitted, and an image (for example, a picture, acharacter, a mark, and the like) showing an operation mode of theelectronic shelf label 1101 is displayed on the monitor 1112.

Note that a configuration of a housing of the electronic shelf label1101 is optional, and is not limited to the example shown in FIG. 28.For example, each of the camera 1111, the monitor 1112, and the modeidentification LED 1113 may be arranged at a position different fromthat in the example of FIG. 28 on a housing of the electronic shelflabel 1101. Furthermore, a plurality of each of the camera 1111, themonitor 1112, and the mode identification LED 1113 may be provided. Thenumber of each of them is optional, and does not need to be the same asone another. Moreover, the electronic shelf label 1101 may include aranging sensor.

Furthermore, the electronic shelf label 1101 has a communicationfunction, and can perform communication with other devices, for example,a server 1103, and the like, via the network 1102.

The network 1102 is an optional communication network, and may be acommunication network for wired communication or a communication networkfor radio communication, or may be configured with both. Furthermore,the network 1102 may be configured with one communication network or aplurality of communication networks. For example, a communicationnetwork and a communication channel of an optional communicationstandard, such as the Internet, a public telephone line network, awide-area communication network for a radio mobile body, such as what iscalled a 3G network, 4G network, or the like, a wide area network (WAN),a local area network (LAN), a radio communication network that performscommunication in compliance with the Bluetooth (registered trademark)standard, a communication channel of short-range radio communication,such as near field communication (NFC) and the like, a communicationchannel of infrared communication, a communication network of wiredcommunication in compliance with standards of a high-definitionmultimedia interface (HDMI), a universal serial bus (USB), and the like,and the like may be included in the network 1102.

The server 1103 provides information associated with a product and thelike to the electronic shelf label 1101. The server 1103 has acommunication function, and can perform communication with otherdevices, such as the electronic shelf label 1101 and the like, via thenetwork 1102. A configuration of the server 1103 will be describedlater. The configuration of the server 1103 is optional, and the server1103 may be configured with one device, configured with a plurality ofdevices, or may be configured as a virtual server that operates on oneor a plurality of servers. Moreover, the configuration may be such thata configuration of the server 1103 is not identified, and what is calleda cloud server is used.

<Installation Example of an Electronic Shelf Label>

FIG. 29 is a diagram showing an example of a state of installation ofthe electronic shelf label 1101. As shown in FIG. 29, the electronicshelf label 1101 is installed in the vicinity of each product type foreach product type. In a case of the example of FIG. 29, the electronicshelf labels 1101-1 to 1101-20 are installed on a front side (customerside) of a product shelf (in other words, a display shelf of a product)1120. A product 1122 is arranged on each of shelves 1121-1 to 1121-4 ofthe product shelf 1120. Note that, hereinafter, the shelves 1121-1 to1121-4 will be referred to as the shelf 1121 in a case where theseshelves do not need to be distinguished from each other. Furthermore, inFIG. 29, only one group of the products 1122 are shown. However, anoptional number of the products 1122 can be arranged at an optionalposition on the shelf 1121.

In FIG. 29, illustration of the product 1122 is omitted. However, inactuality, each of the electronic shelf label 1101 is arranged in thevicinity of a product for which each of the electronic shelf label 1101displays information. For example, information (for example, a productname, a price, and the like) of the product 1122 illustrated in FIG. 29is displayed on the electronic shelf label 1101-1 arranged in thevicinity of the product. A customer can grasp a correspondingrelationship between the product 1122 and the electronic shelf label1101 from their positional relationship, a product name and the likedisplayed on the electronic shelf label.

<Configuration of the Electronic Shelf Label>

FIG. 30 is a block diagram showing a main configuration example of theelectronic shelf label 1101. As shown in FIG. 30, the electronic shelflabel 1101 has processing units such as, for example, an image pickupunit 1131, an image recognition unit 1132, a processing control unit1133, a communication unit 1134, a display control unit 1135, a storageunit 1136, a display unit 1137, and the like.

The image pickup unit 1131 is a processing unit that includes the camera1111 and the like, and picks up an image of an object to obtain apicked-up image (image data). The image pickup unit 1131 supplies imagedata of an obtained picked-up image to the image recognition unit 1132.The image recognition unit 1132 is a processing unit that analyzes imagedata (picked-up image) supplied from the image pickup unit 1131, andrecognizes desired information associated with a product included in thepicked-up image. The image recognition unit 1132 notifies the processingcontrol unit 1133 of a result of the recognition.

The processing control unit 1133 is a processing unit that performsprocessing relating to selection of an operation mode on the basis of aresult of recognition performed by the image recognition unit 1132. Theprocessing control unit 1133 supplies data corresponding to the selectedoperation mode to the communication unit 1134 and the display controlunit 1135. The communication unit 1134 is a processing unit thatperforms communication with other devices. For example, thecommunication unit 1134 supplies data supplied from the processingcontrol unit 1133 to the server 1103 via the network 1102. Furthermore,for example, the communication unit 1134 acquires data from the server1103 via the network 1102, and supplies the data to the display controlunit 1135.

The display control unit 1135 is a processing unit that performsprocessing relating to display control of the display unit 1137. Forexample, the display control unit 1135 generates an image to bedisplayed by using data supplied from the processing control unit 1133,the communication unit 1134, and the like, supplies the image to bedisplayed to the display unit 1137, and causes the display unit 1137 todisplay the image to be displayed. Note that the display control unit1135 can store data supplied from the processing control unit 1133, thecommunication unit 1134, and the like in the storage unit 1136, and readout the data at an optional timing. That is, the display control unit1135 can generate an image to be displayed and cause the display unit1137 to display the image to be displayed at an optional timingregardless of a data supply timing.

The storage unit 1136 is a processing unit that has an optional storagemedium, and can store information supplied from the display control unit1135. The storage unit 1136 supplies stored information to the displaycontrol unit 1135 in response to a request from the display control unit1135. The display unit 1137 is a processing unit that has, for example,the monitor 1112 and the mode identification LED 1113, and performsprocessing relating to display and the like of an image to be displayedsupplied from the display control unit 1135.

FIG. 31 is a block diagram showing a main configuration example of theimage recognition unit 1132 of FIG. 30. As shown in FIG. 31, the imagerecognition unit 1132 has, for example, a specific code recognition unit1141, a customer recognition unit 1142, and a product recognition unit1143.

The specific code recognition unit 1141 performs processing relating torecognition of a specific code for setting an operation mode included ina picked-up image. The customer recognition unit 1142 performsprocessing relating to recognition of a face of a customer and the likeincluded in a picked-up image. The product recognition unit 1143performs processing relating to recognition of information associatedwith a product included in a picked-up image.

<Information Setting Mode>

In a case where displayed information of a product is updated, theelectronic shelf label 1101 operates in an information setting mode. Ina case of the information setting mode, the electronic shelf label 1101picks up an image of information associated with a product by using thecamera 1111 and recognizes the information associated with the productincluded in the picked-up image, so as to update information to bedisplayed on the monitor 1112.

For example, the image pickup unit 1131 picks up an image of informationassociated with a product by using the camera 1111. The informationassociated with a product may be any information, as long as theinformation allows the product to be identified. For example, theinformation may be an appearance, packaging, and the like of a product,or information (identification information) that identifies the productdescribed on places other than the product, such as the packaging, aproduct catalog, and the like. This identification information may beany information, and may be, for example, a product name, a barcode, aQR code (registered trademark), a manufacturing number, anidentification number, and the like, may be information other thanthese, or may be a combination of a plurality of types of information.

The product recognition unit 1143 of the image recognition unit 1132recognizes information associated with a product included in thepicked-up image. The processing control unit 1133 supplies theinformation associated with a product as recognition data recognized bythe image recognition unit 1132 to the communication unit 1134. Thecommunication unit 1134 supplies the recognition data to the server1103. The server 1103 supplies data to be displayed associated with aproduct corresponding to the recognition data to the electronic shelflabel 1101.

The communication unit 1134 of the electronic shelf label 1101 acquiresthe data to be displayed, and the display control unit 1135 generates animage to be displayed by using the data to be displayed. A content ofthe image to be displayed is optional, and, for example, may include atleast any one of a product name, a price, and a barcode (identificationinformation) of a product whose image is picked up by the image pickupunit 1131 (or a product corresponding to “information associated with aproduct” whose image is picked up). Furthermore, for example,information of a production place, a producer, and the like of a productmay be included. Furthermore, information of a discount price and thelike may be included. As a matter of course, information other than theabove examples may be included in the image to be displayed. Forexample, information regarding a product different from a product whoseimage is picked up (or a product corresponding to “informationassociated with a product” whose image is picked up) may be displayed.For example, information associated with a product recommended to acustomer by a seller side may be included. Furthermore, a plurality oftypes of information may be included in the image to be displayed.

The display unit 1137 causes the monitor 1112 to display the image to bedisplayed. That is, the display unit 1137 causes an “image associatedwith a product” corresponding to information associated with a productrecognized by the image recognition unit 1132 to be displayed.

Accordingly, for example, a salesperson of a store that sells a productsets the electronic shelf label 1101 to the “information setting mode”,and causes the camera 1111 to pick up an image of a product and the like(for example, a product, packaging of a product, a product catalog, andthe like). In this manner, a content displayed on the monitor 1112 ofthe electronic shelf label 1101 can be updated to information regardingthe product. That is, dedicated devices, such as a handy terminal,become unnecessary when information is updated, and increase in cost canbe restricted. Furthermore, since operation of dedicated devices, suchas a handy terminal, becomes unnecessary, information to be displayed bythe electronic shelf label 1101 can be updated more easily. That is,usability can be improved.

Note that the configuration may be such that the electronic shelf label1101 operates alone, and information to be displayed can be updatedwithout using the server 1103. For example, the electronic shelf label1101 may store a plurality of pieces of data to be displayed regarding aproduct in advance, and select a piece of data to be displayed to beused on the basis of information associated with a product for which animage is picked up. However, by using the server 1103 as describedabove, limitation caused by a storage capacity of the electronic shelflabel 1101 can be restricted, and more products and more pieces ofinformation can be supported (a more variety of pieces of informationcan be displayed). Furthermore, more up-to-date information can bedisplayed. Moreover, in a case where an entire system is considered,information of a plurality of the electronic shelf labels 1101 can bemanaged collectively by the server 1103. Accordingly, management becomeseasier, and increase in cost can be restricted.

<Information Acquisition Mode>

In a case where information is not updated, the electronic shelf label1101 displays an image to be displayed including information associatedwith a product. At this time, the electronic shelf label 1101 operatesin an information acquisition mode, and can collect informationassociated with a customer. Furthermore, an image to be displayed canalso be updated on the basis of the collected information.

For example, the image pickup unit 1131 picks up an image of a face, ahand, and the like of a customer by using the camera 1111. The customerrecognition unit 1142 of the image recognition unit 1132 recognizesinformation associated with a customer from an image of a customerincluded in the picked-up image. The information associated with acustomer may be any information, and may be, for example, registrationinformation, such as age (or age group), gender, name, contact address,and the like, expression, position, direction, motion, gaze,temperature, state of breathing, lines of palm, fingerprint, and thelike, may be information other than these, or may be a combination of aplurality of types of information.

The processing control unit 1133 supplies the information associatedwith a customer as recognition data recognized by the image recognitionunit 1132 to the communication unit 1134. The communication unit 1134supplies the recognition data to the server 1103. The server 1103acquires and manages the recognition data. In the above manner, theserver 1103 can easily collect information associated with a customerfrom each of the electronic shelf labels 1101, and can analyze a typeand a behavior of a customer and use a result of the analysis and atotal result for planning and the like of a sales strategy. Moreover,the result of the analysis and the total result can also be used forcontrol of information display performed by the electronic shelf label1101. That is, usability can be improved.

Note that, since the electronic shelf label 1101 is installed at aposition close to a product, by operating in the information acquisitionmode as described above, the electronic shelf label 1101 can collectinformation regarding a customer in the vicinity of a product, forexample, which product is compared with which product, whether thecustomer only looks at a product or picks up the product, how manyproducts the customer put in a basket, whether the customer returned theproduct to a shelf, and the like. Accordingly, more useful informationcan be obtained.

Furthermore, for example, the configuration may be such that data to bedisplayed on the electronic shelf label 1101 is selected on the basis ofinformation associated with a customer collected by the server 1103, andthe data to be displayed is supplied to the electronic shelf label 1101.A content of the data to be displayed is optional. For example,information associated with a product may be included in the data to bedisplayed. For example, the server 1103 may select, as data to bedisplayed, information associated with a product that is expected tohave an advertising effect by being presented to a customer on the basisof collected information associated with the customer. Furthermore, forexample, the server 1103 may identify a way of displaying by which amore significant effect can be obtained (for example, a layout of animage to be displayed and the like) on the basis of collectedinformation associated with a customer, and select data to be displayedthat shows the displaying method.

The communication unit 1134 of the electronic shelf label 1101 acquiresthe data to be displayed, the display control unit 1135 generates animage to be displayed (image relating to a product) from the data to bedisplayed, and the display unit 1137 causes the monitor 1112 to displaythe image to be displayed. That is, the display unit 1137 can cause themonitor 1112 to display an image relating to a product corresponding tocollected information associated with a customer. Accordingly, theelectronic shelf label 1101 can perform display by which a moresignificant effect (for example, a sales promotion effect) can beexpected.

<Switching of Modes>

The electronic shelf label 1101 can switch between the informationsetting mode and the information acquisition mode described above. Thisswitching method is optional. For example, the configuration may be suchthat a switch for switching an operation mode is provided on a housingof the electronic shelf label 1101, and the user and the like manuallymoves the switch to switch an operation mode. Furthermore, for example,the configuration may be such that the electronic shelf label 1101 picksup an image of a specific code so that the switching of an operationmode can be performed.

For example, the image pickup unit 1131 uses the camera 1111 to pick upan image (of a specific code) exclusively used for switching anoperation mode. This specific code may be any image. The specific codemay be a character, a number, a mark, or the like, or may be a barcode,a QR code, or the like. The specific code may be a face, a hand (a veinand a fingerprint), and the like of a person registered in advance, suchas an administrator or the like. Furthermore, the specific code may besomething other than the above, or a combination of a plurality of typesof information.

The specific code recognition unit 1141 of the image recognition unit1132 recognizes a specific code included in the picked-up image. Theprocessing control unit 1133 switches an operation mode in accordancewith a result of recognition of the specific code. In other words, theprocessing control unit 1133 selects information to be used asrecognition data in accordance with a result of recognition of aspecific code. In other words, the processing control unit 1133 controlswhether to supply information associated with a product to the server1103, or to supply information associated with a customer to the server1103.

The display unit 1137 updates display of the mode identification LED1113 in accordance with the switching of a mode. For example, thedisplay unit 1137 switches a display color of the mode identificationLED 1113, or lights up, turns off, or blinks the mode identification LED1113. Note that the display unit 1137 may use the mode identificationLED 1113 to notify that an operation mode is switched.

As described above, the electronic shelf label 1101 can easily switch anoperation mode without using a dedicated device. That is, the electronicshelf label 1101 can operate in the two modes described above moreeasily. Accordingly, usability can be improved.

<Configuration of a Server>

FIG. 32 is a block diagram showing a main configuration example of theserver 1103. As shown in FIG. 32, the server 1103 has a centralprocessing unit (CPU) 1151, a read only memory (ROM) 1152, and a randomaccess memory (RAM) 1153 which are mutually connected via a bus 1154.

Furthermore, an input and output interface 1160 is also connected to thebus 1154. An input unit 1161, an output unit 1162, and a storage unit1163, a communication unit 1164, and a drive 1165 are connected to theinput and output interface 1160.

The input unit 1161 has an optional input device such as, for example, akeyboard, a mouse, a touch panel, an image sensor, a microphone, aswitch, an input terminal, and the like. The output unit 1162 has anoptional output device such as, for example, a display, a speaker, anoutput terminal, and the like. The storage unit 1163 has an optionalstorage medium such as, for example, a non-volatile memory and the like,such as a hard disk, a RAM disk, a solid state drive (SSD), a USBmemory, and the like. The communication unit 1164 has a communicationinterface of an optional communication standard for wired or radiocommunication, or both, such as, for example, Ethernet (registeredtrademark), Bluetooth, USB, HDMI (registered trademark), IrDA, and thelike. The drive 1165 drives a removable medium 1171 having an optionalstorage medium, such as a magnetic disk, an optical disk, amagneto-optical disk, a semiconductor memory, or the like, that ismounted on itself.

In the server 1103 configured as described above, for example, the CPU1151 loads a program stored in the storage unit 1163 into the RAM 1153via the input and output interface 1160 and the bus 1154 and executesthe program so as to be able to perform a function described later.Furthermore, data and the like necessary for the CPU 1151 to execute avariety of types of processing are stored as appropriate in the RAM1153.

A program executed by the CPU 1151 can be recorded in the removablemedium 1171 used, for example, as a package medium and the like, andapplied. In such a case, the program can be installed in the storageunit 1163 via the input and output interface 1160 by mounting theremovable medium 1171 on the drive 1165. Furthermore, the program canalso be provided via a wired or radio transmission medium, such as alocal area network, the Internet, or digital satellite broadcasting. Insuch a case, the program can be received by the communication unit 1164and installed in the storage unit 1163. Other than the above, theprogram can also be installed in the ROM 1152 or the storage unit 1163in advance.

<Process of Information Acquisition Processing>

Next, processing executed in the electronic shelf label system 1100 asdescribed above will be described. The electronic shelf label 1101executes information acquisition processing to perform processing in theinformation setting mode described above, processing in the informationacquisition mode, switching of an operation mode, or the like.

An example of a process of the information acquisition processing willbe described with reference to a flowchart of FIG. 33. When theinformation acquisition processing is started, the image pickup unit1131 of the electronic shelf label 1101 starts image pickup by thecamera 1111 in step S1101. Furthermore, in step S1102, each processingunit of the electronic shelf label 1101 performs initial setting.

In step S1103, the image pickup unit 1131 picks up an image of an objectby using the camera 1111 to obtain a picked-up image. In step S1104, theimage recognition unit 1132 performs image recognition processing torecognize information included in the picked-up image. Details of theimage recognition processing will be described later. In step S1105, theprocessing control unit 1133 performs recognition data selectionprocessing on the basis of a result of the image recognition processingto select data to be used as recognition data. Details of therecognition data selection processing will be described later. In stepS1106, the display control unit 1135 controls the display unit 1137 toperform display that shows a current operation mode. The display unit1137 performs the display by using the mode identification LED 1113. Instep S1107, the communication unit 1134 transmits the recognition dataselected by the processing of step S1105 to the server 1103.

In step S1108, the electronic shelf label 1101 determines whether or notto finish the information acquisition processing, and, if determiningnot to finish the processing, the electronic shelf label 1101 returnsthe processing to step S1103 and repeats the processing of step S1103and subsequent steps. Furthermore, when the electronic shelf label 1101determines to finish the information acquisition processing in stepS1108, the information acquisition processing is finished.

<Process of Image Recognition Processing>

Next, an example of a process of image recognition processing executedin step S1104 in FIG. 33 will be described with reference to flowchartsof FIGS. 34 and 35.

When the image recognition processing is started, the productrecognition unit 1143 of the image recognition unit 1132 analyzes apicked-up image, and recognizes information associated with a productincluded in the picked-up image. For example, the product recognitionunit 1143 recognizes a design of a housing and packaging of a product, aproduct name, identification information, and the like included in thepicked-up image. The product recognition unit 1143 stores a result ofthe recognition of information associated with a product.

Furthermore, the customer recognition unit 1142 of the image recognitionunit 1132 analyzes a picked-up image, and recognizes informationassociated with a customer included in the picked-up image. For example,the customer recognition unit 1142 recognizes a face, a hand, a motion,a gaze, and the like of a customer included in a picked-up image. Thecustomer recognition unit 1142 stores a result of the recognition ofinformation associated with a customer.

Furthermore, the specific code recognition unit 1141 of the imagerecognition unit 1132 analyzes a picked-up image, and recognizes aspecific code included in the picked-up image. For example, the specificcode recognition unit 1141 recognizes a barcode, a QR code, anidentification number, a face of an administrator, and the like includedin a picked-up image.

Each piece of the above processing may be executed in parallel as shownin steps S1111 to S1113 in FIG. 34, or may be executed in series asshown in steps S1121 to S1123 in FIG. 35. When the above processing isfinished, the image recognition processing is finished, and theprocessing returns to FIG. 33.

<Process of Recognition Data Selection Processing>

Next, an example of a process of recognition data selection processingexecuted in step S1105 in FIG. 33 will be described with reference toflowcharts of FIGS. 36 to 38.

For example, if an administrator positions a predetermined specific code(for example, optional information, such as a barcode, an identificationnumber, a face of the administrator, or the like) in front of the camera1111 and causes the camera 1111 to pick up an image, the specific coderecognition unit 1141 recognizes the specific code in step S1104 in FIG.33. The specific code used for designation of an operation mode may be,for example, information unique to a mode. That is, the configurationmay be such that a first specific code showing an information settingmode and a second specific code showing an information acquisition modeexist as the specific codes, and an operation mode is controlleddepending on of which of the specific codes an image is picked up. Forexample, the configuration may be such that, in a case where an image ofthe first specific code is picked up, the information setting mode isselected, and in a case where an image of the second specific code ispicked up, the information acquisition mode is selected. The firstspecific code and the second specific code may be any information aslong as the codes are distinguishable from each other.

An example of a process of the recognition data selection processing inthis case will be described with reference to a flowchart of FIG. 36.When the recognition data selection processing is started, theprocessing control unit 1133 determines, in step S1131, whether or notthe first specific code is recognized by the processing of step S1104 inFIG. 33. In a case where the first specific code is determined to berecognized, the processing proceeds to step S1132.

In step S1132, the processing control unit 1133 sets an operation modeto the information setting mode, and reads out stored data of a resultof recognition of information associated with a product as recognitiondata from the product recognition unit 1143. The readout result ofrecognition of information associated with a product is transmitted asrecognition data to the server 1103 in step S1107 in FIG. 33.Furthermore, in this case, in step S1106 in FIG. 33, display of the modeidentification LED 1113 is set to display showing the informationsetting mode. When the processing of step S1132 is finished, therecognition data selection processing is finished, and the processingreturns to FIG. 33.

In step S1131, in a case where the first specific code is determined notto be recognized, the processing proceeds to step S1133. In step S1133,the processing control unit 1133 determines whether or not the secondspecific code is recognized by the processing of step S1104 in FIG. 33.In a case where the second specific code is determined to be recognized,the processing proceeds to step S1134.

In step S1134, the processing control unit 1133 sets an operation modeto the information acquisition mode, and reads out stored data of aresult of recognition of information associated with a customer asrecognition data from the customer recognition unit 1142. The readoutresult of recognition of information associated with a customer istransmitted as recognition data to the server 1103 in step S1107 in FIG.33. Furthermore, in this case, in step S1106 in FIG. 33, display of themode identification LED 1113 is set to display showing the informationacquisition mode. When the processing of step S1134 is finished, therecognition data selection processing is finished, and the processingreturns to FIG. 33.

In step S1133, in a case where the second specific code is determinednot to be recognized, the processing proceeds to step S1135. In stepS1135, the processing control unit 1133 keeps an operation mode to themode that is the same as that when the stored data is read outpreviously, and reads out stored data of information corresponding tothe mode as recognition data. For example, in a case where the previousmode is the information setting mode, the processing control unit 1133reads out, in a manner similar to the case of step S1132, stored data ofa result of recognition of information associated with a product asrecognition data from the product recognition unit 1143. Furthermore,for example, in a case where the previous mode is the informationacquisition mode, the processing control unit 1133 reads out, in amanner similar to the case of step S1134, stored data of a result ofrecognition of information associated with a customer as recognitiondata from the customer recognition unit 1142. The readout recognitiondata is transmitted to the server 1103 in step S1107 in FIG. 33.Furthermore, in this case, in step S1106 in FIG. 33, display of the modeidentification LED 1113 is kept to be the previous one. When theprocessing of step S1135 is finished, the recognition data selectionprocessing is finished, and the processing returns to FIG. 33.

Furthermore, the configuration may be such that a single specific codeis used for designation of an operation mode, and an operation mode isswitched in a case where the specific code is recognized. An example ofa process of the recognition data selection processing in this case willbe described with reference to a flowchart of FIG. 37. When therecognition data selection processing is started, the processing controlunit 1133 determines, in step S1141, whether or not the specific code isrecognized by the processing of step S1104 in FIG. 33. In a case wherethe specific code is determined not to be recognized, the processingproceeds to step S1142.

In step S1142, the processing control unit 1133 sets an operation modeto the mode that is the same as that when read out previously, and readsout stored data read out previously as recognition data. For example, ifa previous operation mode is the information setting mode, and storeddata of a result of recognition of information associated with a productis read out as recognition data, the processing control unit 1133 setsan operation mode to the information setting mode this time again, andreads out stored data of a result of recognition of informationassociated with a product as recognition data. In contrast, if aprevious operation mode is the information acquisition mode, and storeddata of a result of recognition of information associated with acustomer is read out as recognition data, the processing control unit1133 sets an operation mode to the information acquisition mode thistime again, and reads out stored data of a result of recognition ofinformation associated with a customer as recognition data. When theprocessing of step S1142 is finished, the recognition data selectionprocessing is finished, and the processing returns to FIG. 33.

Furthermore, in step S1141, in a case where the specific code isdetermined to be recognized, the processing proceeds to step S1143. Instep S1143, the processing control unit 1133 switches an operation mode,and reads out stored data that is not read out previously as recognitiondata. For example, if a previous operation mode is the informationsetting mode, and stored data of a result of recognition of informationassociated with a product is read out as recognition data, theprocessing control unit 1133 switches an operation mode to theinformation acquisition mode this time, and reads out stored data of aresult of recognition of information associated with a customer asrecognition data. In contrast, if a previous operation mode is theinformation acquisition mode, and stored data of a result of recognitionof information associated with a customer is read out as recognitiondata, the processing control unit 1133 switches an operation mode to theinformation setting mode this time, and reads out stored data of aresult of recognition of information associated with a product asrecognition data. When the processing of step S1143 is finished, therecognition data selection processing is finished, and the processingreturns to FIG. 33.

Furthermore, the configuration may be such that, in a case where eitherone of the information setting mode and the information acquisition modeis set in an initial state, and the specific code is recognized, anoperation mode is set to the other mode for a certain period of time. Anexample of a process of the recognition data selection processing inthis case will be described with reference to a flowchart of FIG. 38.Note that, here, description will be made by assuming that theinformation acquisition mode is set in an initial state. When therecognition data selection processing is started, the processing controlunit 1133 determines, in step S1151, whether or not the specific code isrecognized by the processing of step S1104 in FIG. 33. In a case wherethe specific code is determined to be recognized, the processingproceeds to step S1152.

In step S1152, the processing control unit 1133 switches an operationmode to the information setting mode, and reads out stored data of aresult of recognition of information associated with a product asrecognition data. When the processing of step S1152 is finished, therecognition data selection processing is finished, and the processingreturns to FIG. 33.

Furthermore, in step S1151, in a case where the specific code isdetermined not to be recognized, the processing proceeds to step S1153.In step S1153, the processing control unit 1133 determines whether ornot a predetermined period of time or longer has elapsed since thespecific code is recognized previously. In a case where a predeterminedperiod of time or longer is determined not to have elapsed, theprocessing returns to step S1152.

Furthermore, in step S1153, in a case where a predetermined period oftime or longer is determined to have elapsed since the specific code isrecognized previously, the processing proceeds to step S1154. In stepS1154, the processing control unit 1133 sets an operation mode to theinformation acquisition mode, and reads out stored data of a result ofrecognition of information associated with a customer as recognitiondata. When the processing of step S1154 is finished, the recognitiondata selection processing is finished, and the processing returns toFIG. 33.

<Process of Data Provision Processing>

The server 1103, to which recognition data is supplied from theelectronic shelf label 1101 as described above, performs data provisionprocessing so as to provide data to be displayed corresponding to therecognition data to the electronic shelf label 1101. An example of aprocess of the data provision processing will be described withreference to a flowchart of FIG. 39.

When the data provision processing is started, the CPU 1151 of theserver 1103, in step S1161, controls the communication unit 1164 toacquire recognition data supplied from the electronic shelf label 1101.In step S1162, the CPU 1151 stores the recognition data acquired by theprocessing of step S1161 in the storage unit 1163. In step S1163, theCPU 1151 selects data to be displayed (data including information to bedisplayed on the electronic shelf label 1101) corresponding to therecognition data among a group of pieces of data to be displayed storedin the storage unit 1163. In step S1164, the CPU 1151 controls thecommunication unit 1164 to supply the selected data to be displayed tothe electronic shelf label 1101.

When the processing of step S1164 is finished, the data provisionprocessing is finished.

<Process of Display Control Processing>

When data to be displayed is supplied from the server 1103 as describedabove, the electronic shelf label 1101 executes display controlprocessing, and causes the monitor 1112 to display the data to bedisplayed. An example of a process of the display control processingwill be described with reference to a flowchart of FIG. 40.

When the display control processing is started, the communication unit1134, in step S1171, acquires data to be displayed supplied from theserver 1103. In step S1172, the display control unit 1135 uses the datato be displayed to generate an image to be displayed. In step S1173, thedisplay unit 1137 causes the monitor 1112 to display the image to bedisplayed.

When the processing of step S1173 is finished, the display controlprocessing is finished.

Note that the configuration may be such that the data to be displayedacquired in step S1171 is temporarily stored in the storage unit 1136,and, at a timing of displaying, the display control unit 1135 reads outthe data to be displayed, and generates an image to be displayed. In theabove manner, a display timing of an image to be displayed can be madeindependent of an acquisition timing of data to be displayed.

By performing each piece of processing as described above, theelectronic shelf label 1101 can operate in the information setting modeand the information acquisition mode described above, and, furthermore,these operation modes can be switched, and usability can be improved.

3. Third Embodiment

<Selection by Electronic Shelf Label>

The second embodiment describes that information (data to be displayed)displayed by the electronic shelf label 1101 is selected by the server1103. However, the information may be selected by the electronic shelflabel 1101. For example, the configuration may be such that the server1103 supplies a plurality of pieces of data to be displayed to theelectronic shelf label 1101 in advance, and the electronic shelf label1101 selects and displays data corresponding to information associatedwith a product recognized from a picked-up image from among the supplieddata to be displayed.

<Process of Data Provision Processing>

In such a case, the server 1103 executes the data provision processingto supply data to be displayed to the electronic shelf label 1101. Anexample of a process of the data provision processing will be describedwith reference to a flowchart of FIG. 41.

When the data provision processing is started, the CPU 1151 of theserver 1103 selects, in step S1201, a plurality of pieces of data fromdata to be displayed stored in the storage unit 1163. The standard ofthis selection is optional. In step S1202, the CPU 1151 controls thecommunication unit 1164 to supply a plurality of pieces of the data tobe displayed selected in step S1201 to the electronic shelf label 1101.

When the processing of step S1202 is finished, the data provisionprocessing is finished.

<Process of Acquisition Processing for Data to be Displayed>

With respect to this processing, the electronic shelf label 1101executes acquisition processing for data to be displayed so as toacquire data to be displayed supplied from the server 1103. An exampleof a process of the acquisition processing for data to be displayed willbe described with reference to a flowchart of FIG. 42.

When the acquisition processing for data to be displayed is started, thecommunication unit 1134 of the electronic shelf label 1101 acquires, instep S1211, a plurality of pieces of data to be displayed supplied fromthe server 1103. In step S1212, the storage unit 1136 stores a pluralityof pieces of data to be displayed acquired in step S1211.

When the processing of step S1212 is finished, the acquisitionprocessing for data to be displayed is finished.

<Process of Information Acquisition Processing>

The electronic shelf label 1101 that has performed the acquisitionprocessing for data to be displayed executes information acquisitionprocessing to perform processing in the information setting modedescribed above, processing in the information acquisition mode,switching of an operation mode, or the like.

An example of a process of the information acquisition processing inthis case will be described with reference to a flowchart of FIG. 43.When the information acquisition processing is started, the image pickupunit 1131 of the electronic shelf label 1101 starts image pickup byusing the camera 1111 in step S1221. Furthermore, in step S1222, eachprocessing unit of the electronic shelf label 1101 performs initialsetting.

In step S1223, the image pickup unit 1131 picks up an image of an objectby using the camera 1111 to obtain a picked-up image. In step S1224, theimage recognition unit 1132 performs image recognition processing torecognize information included in the picked-up image. Details of theimage recognition processing are optional. For example, processingsimilar to that in the case of the second embodiment may be executed.

In step S1225, the processing control unit 1133 performs recognitiondata selection processing on the basis of a result of the imagerecognition processing to select data to be used as recognition data.Details of the recognition data selection processing are optional. Forexample, processing similar to that in the case of the second embodimentmay be executed.

In step S1226, the display control unit 1135 controls the display unit1137 to perform display that shows a current operation mode. The displayunit 1137 performs the display by using the mode identification LED1113. In step S1227, the electronic shelf label 1101 performs displaycontrol processing to cause the monitor 1112 to display informationcorresponding to the recognition data selected by the processing of stepS1225. Details of the display control processing will be describedlater.

In step S1228, the communication unit 1134 transmits the recognitiondata selected by the processing of step S1225 to the server 1103.

In step S1229, the electronic shelf label 1101 determines whether or notto finish the information acquisition processing, and, if determiningnot to finish the processing, the electronic shelf label 1101 returnsthe processing to step S1223 and repeats the processing of step S1223and subsequent steps. Furthermore, when the electronic shelf label 1101determines to finish the information acquisition processing in stepS1229, the information acquisition processing is finished.

<Process of Display Control Processing>

Next, an example of a process of the display control processing executedin step S1227 in FIG. 43 will be described with reference to a flowchartof FIG. 44. When the display control processing is started, the displaycontrol unit 1135 selects, in step S1231, data to be displayedcorresponding to the recognition data selected by the processing of stepS1225 in FIG. 43 from among a plurality of pieces of data to bedisplayed stored in the storage unit 1136, and reads out the data to bedisplayed from the storage unit 1136.

In step S1232, the display control unit 1135 uses the readout data to bedisplayed to generate an image to be displayed, and supplies the imageto be displayed to the display unit 1137. In step S1233, the displayunit 1137 causes the monitor 1112 to display the image to be displayedgenerated by the processing of step S1232.

When the processing of step S1233 is finished, the display controlprocessing is finished, and the processing returns to FIG. 43.

<Process of Data Acquisition Processing>

With respect to the information acquisition processing as describedabove, the server 1103 executes data acquisition processing so as tocollect recognition data supplied from the electronic shelf label 1101.An example of a process of the data acquisition processing will bedescribed with reference to a flowchart of FIG. 45.

When the data acquisition processing is started, the CPU 1151 of theserver 1103 controls, in step S1241, the communication unit 1164 toacquire recognition data supplied from the electronic shelf label 1101.In step S1242, the CPU 1151 stores the acquired recognition data in thestorage unit 1163. When the recognition data is stored in the storageunit 1163, the data acquisition processing is finished.

By executing each piece of processing as described above, the electronicshelf label 1101 can easily select data to be displayed corresponding toinformation associated with a product or a customer recognized in apicked-up image, and display the image to be displayed. In the abovemanner, for example, even in a case where communication with the server1103 is temporarily interrupted, the electronic shelf label 1101 cancause the monitor 1112 to display an image to be displayed correspondingto recognized information. That is, usability can be improved.

4. Fourth Embodiment

<Identification of Size of a Face, and the Like>

In the above description, information associated with a customer iscollected. This information associated with a customer may be anyinformation. For example, the configuration may be such that theinformation includes at least any one of a position of a face of acustomer (how close the customer approaches a product), a time at whicha face of a customer is detected (when the customer comes), and a timeperiod in which a face of a customer is detected (how long the customeris there). That is, information showing an action of a customer whichshows how much the customer shows an interest in a product may beincluded in information associated with a customer.

For example, a position of a face of a customer may be determined on thebasis of size of a detected face (that is, how close the customerapproaches a product may be determined). For example, the configurationmay be such that, as shown in A of FIG. 46, the electronic shelf label1101 sets frames 1311 and 1312 for an angle of view of a picked-up image1301 (that is, an image pickup range of the camera 1111), and determinessize of a face by comparing these two frames with a face image. Notethat, as shown in A of FIG. 46, the frame 1311 is set to be smaller thanthe frame 1312. Furthermore, as shown in A of FIG. 46, the frames 1311and 1312 may be set around a center of an angle of view of the picked-upimage 1301 or may be movable. For example, the frames 1311 and 1312 maybe set to a position corresponding to a position of a face image in thepicked-up image 1301. Furthermore, size and shapes of the frames 1311and 1312 may be changed in accordance with a situation.

For example, as shown in B of FIG. 46, when a face image 1321 is smallerthan the frame 1311, the electronic shelf label 1101 determines that thecustomer is positioned far from a product (the electronic shelf label1101), and estimates that an interest of the customer on the product issmall. Furthermore, for example, as shown in C of FIG. 46, in a casewhere the face image 1321 is larger than the frame 1311, and is smallerthan the frame 1312, the electronic shelf label 1101 determines that thecustomer is positioned closer to a product (the electronic shelf label1101) than the case of B of FIG. 46, and estimates that an interest ofthe customer on the product is higher than that in the case of B of FIG.46. Moreover, for example, as shown in D of FIG. 46, in a case where theface image 1321 is larger than the frame 1312, the electronic shelflabel 1101 determines that the customer is positioned closer to aproduct (the electronic shelf label 1101) than in the case of C of FIG.46, and estimates that an interest of the customer on the product ishigher than that in the case of C of FIG. 46.

<Process of Customer Information Recognition Processing>

In this case, the customer recognition unit 1142 performs customerinformation recognition processing in step S1104 in FIG. 33 (step S1112in FIG. 34 or step S1122 in FIG. 35), so as to recognize theseparameters.

An example of a process of the customer information recognitionprocessing will be described with reference to a flowchart of FIG. 47.When the customer information recognition processing is started, thecustomer recognition unit 1142 detects a face from a picked-up image instep S1301. In step S1302, the customer recognition unit 1142 recognizesa face detection time that is a time at which a face is detected in stepS1301. That is, the customer recognition unit 1142 recognizes when acustomer shows an interest in a product. In step S1303, the customerrecognition unit 1142 determines and recognizes size of the facedetected in step S1301 by, for example, the method described withreference to FIG. 46. That is, the customer recognition unit 1142recognizes how much a customer shows an interest. In step S1304, thecustomer recognition unit 1142 measures and recognizes a time period inwhich a face is detected in step S1301. That is, the customerrecognition unit 1142 recognizes how long a customer shows an interest.Then, the customer recognition unit 1142 stores these results of therecognition of information associated with a customer.

When the processing of step S1304 is finished, the customer informationrecognition processing is finished, and the processing returns to FIG.33.

By performing the customer information recognition processing asdescribed above, the electronic shelf label 1101 can obtain a morevariety of information as information associated with a customer.

5. Fifth Embodiment

<Moving Object Detection>

Furthermore, the electronic shelf label 1101 may recognize a motion(behavior) of a customer as information associated with a customer.Moreover, in such a case, a camera for detecting a motion of a customermay be provided separately from the camera 1111. For example, as shownin A of FIG. 48, a camera 1341 separate from the camera 1111 may beprovided in a housing of the electronic shelf label 1101. In this case,the configuration may be such that, for example, as shown in B of FIG.48, the electronic shelf label 1101 picks up an image of a face 1351 ofa customer by using the camera 1111, and detects a motion of a hand 1352of the customer who attempts to touch a product 1122-1 or a product1122-2 placed on the shelf 1121 by using the camera 1341.

Note that, in a case of the example of FIG. 48, the camera 1341 isprovided on a top surface (a surface facing an upper side) of a housingof the electronic shelf label 1101. A position and a shooting directionof the camera 1341 are optional. Furthermore, performance for resolutionrequired for the camera 1341, which only needs to obtain a picked-upimage from which a moving object can be detected, is lower than thatrequired for the camera 1111. That is, the camera 1341 may use an imagepickup element of a lower resolution than the camera 1111. Furthermore,in order to allow a moving object of higher speed to be detected, aframe rate of the camera 1341 may be set to be higher than that of thecamera 1111.

<Configuration of the Electronic Shelf Label>

A main configuration example of the electronic shelf label 1101 in thiscase is shown in a block diagram of FIG. 49. As shown in FIG. 49, inthis case, the electronic shelf label 1101 further has an image pickupunit 1361 and a moving object detection unit 1362 in addition to theconfiguration shown in FIG. 30, and has a processing control unit 1363in place of the processing control unit 1133.

The image pickup unit 1361 is a processing unit that includes a camera1341 and the like, and picks up an image of an object to obtain apicked-up image (image data). The image pickup unit 1361 supplies imagedata of an obtained picked-up image to the moving object detection unit1362. The moving object detection unit 1362 is a processing unit thatanalyzes image data (a picked-up image) supplied from the image pickupunit 1361, and detects a moving object (for example, a hand of acustomer, and the like) included in the picked-up image. This detectionmethod of a moving object is optional. For example, the configurationmay be such that a moving object detection is performed by comparingimages between a current frame and a previous frame. Furthermore, forexample, the configuration may be such that a background image (an imagein an initial state) is set, and a moving object detection is performedby comparing the background image and an image of a current frame. Themoving object detection unit 1362 notifies the processing control unit1363 of a result of the detection.

The processing control unit 1363 is a processing unit that performsprocessing relating to selection of an operation mode on the basis of aresult of recognition performed by the image recognition unit 1132 and aresult of detection performed by the moving object detection unit 1362.The processing control unit 1363 supplies data corresponding to theselected operation mode to the communication unit 1134 and the displaycontrol unit 1135. For example, in a case where the “informationacquisition mode” is selected, the processing control unit 1363 selectsa result of recognition performed by the image recognition unit 1132 anda result of moving object detection performed by the moving objectdetection unit 1362 as recognition data, and supplies the recognitiondata to the communication unit 1134. That is, in a case of theinformation acquisition mode, the communication unit 1134 suppliesrecognition data including a result of recognition performed by theimage recognition unit 1132 and a result of moving object detection (forexample, whether or not a customer picks up a product, and the like)performed by the moving object detection unit 1362 to the server 1103via the network 1102.

<Process of Customer Information Recognition Processing>

In this case, in step S1101 of the information acquisition processing ofFIG. 33 (step S1221 in a case of the information acquisition processingof FIG. 43), the image pickup unit 1361 starts image pickup togetherwith the image pickup unit 1131, and, in step S1103 (step S1223 in acase of the information acquisition processing of FIG. 43), acquires apicked-up image together with the image pickup unit 1131.

The customer recognition unit 1142 and the moving object detection unit1362 perform customer information recognition processing in step S1104in FIG. 33 (step S1112 in FIG. 34 or step S1122 in FIG. 35), so as torecognize information associated with a customer.

An example of a process of the customer information recognitionprocessing will be described with reference to a flowchart of FIG. 50.When the customer information recognition processing is started, eachpiece of processing from step S1321 to step S1324 is executed in asimilar manner as each piece of processing from step S1301 to step S1304in FIG. 47. In step S1325, the moving object detection unit 1362 detectsa moving object from a picked-up image obtained by the image pickup unit1361. That is, the moving object detection unit 1362 recognizes whetheror not a customer picks up a product. Then, the customer recognitionunit 1142 stores these results of the recognition of informationassociated with a customer.

When the processing of step S1325 is finished, the customer informationrecognition processing is finished, and the processing returns to FIG.33. By performing the customer information recognition processing asdescribed above, the electronic shelf label 1101 can obtain a morevariety of information as information associated with a customer.

6. Sixth Embodiment

<Display Control>

The fifth embodiment is described with respect to the case whereinformation associated with a motion (behavior) of a customer iscollected as information associated with the customer. The configurationmay be such that information (an image) displayed on the electronicshelf label 1101 is controlled on the basis of information associatedwith the motion (behavior) of a customer. For example, the configurationmay be such that the server 1103 selects data to be displayed that issupplied to the electronic shelf label 1101 on the basis of informationassociated with a motion (behavior) of a customer supplied asrecognition data. In such a case, the server 1103 preferably executesdata provision processing in a similar manner as in the case describedin the second embodiment, selects data to be displayed in accordancewith recognition data (information associated with a motion (behavior)of a customer), and supplies the recognition data to the electronicshelf label 1101. Furthermore, for example, the configuration may besuch that the electronic shelf label 1101 selects data to be displayedon the basis of information associated with a recognized motion(behavior) of a customer. In such a case, the electronic shelf label1101 preferably executes display control processing in a similar manneras in the case described in the third embodiment, selects data to bedisplayed in accordance with information associated with a motion(behavior) of a customer, and displays the recognition data.

In the above manner, for example, when a customer approaches theelectronic shelf label 1101 within a predetermined distance, theelectronic shelf label 1101 can change a displayed content of themonitor 1112. For example, the electronic shelf label 1101 can performdisplay in more varieties and more effectively in accordance with asituation, in such a manner as causing the monitor 1112 to displayservice information that introduces a service targeted for the customer,and the like, for example.

<Attributes of a Customer>

Furthermore, for example, the configuration may be such that information(an image) to be displayed on the electronic shelf label 1101 iscontrolled in accordance with attributes of a customer, such as age (orage group), gender, and the like of the customer. In such a case, thecustomer recognition unit 1142 determines attributes of a customerincluded in a picked-up image, and recognizes a result of thedetermination as information associated with the customer. Processingother than the above may be executed in a similar manner as the casesdescribed in the second and third embodiments.

<Process of Customer Information Recognition Processing>

In this case, an example of a process of the customer informationrecognition processing that the customer recognition unit 1142 executesin step S1104 in FIG. 33 (step S1112 in FIG. 34 or step S1122 in FIG.35) will be described with reference to a flowchart of FIG. 51.

When the customer information recognition processing is started, eachpiece of processing from step S1341 to step S1344 is executed in asimilar manner as each piece of processing from step S1301 to step S1304in FIG. 47. In step S1345, the customer recognition unit 1142 determinesattributes of a customer, such as gender, age, and the like of thecustomer, from a face detected in step S1341. Then, the customerrecognition unit 1142 stores the result of the recognition and result ofdetermination of information associated with a customer.

When the processing of step S1345 is finished, the customer informationrecognition processing is finished, and the processing returns to FIG.33. By performing the customer information recognition processing asdescribed above, the electronic shelf label 1101 can obtain a morevariety of information as information associated with a customer.Accordingly, the electronic shelf label 1101 can control a displaycontent in accordance with more varieties of information. That is, theelectronic shelf label 1101 can display more varieties of information.

<Identification Information of a Customer>

Furthermore, for example, the configuration may be such that information(an image) to be displayed on the electronic shelf label 1101 iscontrolled in accordance with identification information of a customer.For example, the configuration may be such that, if a customer whoapproaches the electronic shelf label 1101 holds an ID card or the likein front of the camera 1111 of the electronic shelf label 1101, andcauses an image of a customer-specific ID code registered in (displayedon) the card to be picked up, the electronic shelf label 1101 displaysinformation corresponding to the customer-specific ID code (in otherwords, information corresponding to the customer who is the owner of theID card). For example, the configuration may be such that, in a casewhere the customer is a special customer (regular customer), the monitor1112 displays a price applied with a predetermined discount in place ofa regular price of a product. Processing other than the above may beexecuted in a similar manner as the cases described in the second andthird embodiments.

By using the above identification information, more detailedidentification of a customer can be performed more precisely and moreeasily. Accordingly, more appropriate information can be presented toeach customer. That is, display control can be performed more in detail.

<Configuration of the Image Recognition Unit>

A main configuration example of the image recognition unit 1132 in thiscase is shown in FIG. 52. In this case, the image recognition unit 1132has a customer-specific ID code recognition unit 1471 in addition to theconfiguration shown in FIG. 31. The customer-specific ID coderecognition unit 1471 performs processing relating to recognition of acustomer-specific ID code included in a picked-up image.

<Process of Image Recognition Processing>

An example of a process of the image recognition processing executed instep S1104 in FIG. 33 in this case will be described with reference to aflowchart of FIG. 53. In this case, when the image recognitionprocessing is started, the product recognition unit 1143 recognizesinformation associated with a product in step S1361. In step S1362, thecustomer recognition unit 1142 recognizes information associated with acustomer. In step S1363, the customer-specific ID code recognition unit1471 recognizes a customer-specific ID code included in a picked-upimage. In step S1364, the specific code recognition unit 1141 recognizesa specific code. When the processing of step S1364 is finished, theinformation recognition processing is finished, and the processingreturns to FIG. 33.

Note that, in this case as well, processing of each step of the imagerecognition processing can be executed in parallel with one another asin the example of FIG. 34.

<Process of Recognition Data Selection Processing>

Next, an example of a process of the recognition data selectionprocessing executed in step S1105 in FIG. 33 will be described withreference to a flowchart of FIG. 54. In this case, when the recognitiondata selection processing is started, each piece of processing from stepS1371 to step S1373 is executed in a similar manner as each piece ofprocessing from step S1131 to step S1133 in FIG. 36.

If the second specific code is recognized, in step S1374, the processingcontrol unit 1133 sets an operation mode to the information acquisitionmode, and reads out stored data of a result of recognition ofinformation associated with a customer and stored data of a result ofrecognition of a customer-specific ID code as recognition data from thecustomer recognition unit 1142. The readout result of recognition ofinformation associated with a customer is transmitted as recognitiondata to the server 1103 in step S1107 in FIG. 33. Furthermore, in thiscase, in step S1106 in FIG. 33, display of the mode identification LED1113 is set to display showing the information acquisition mode. Whenthe processing of step S1374 is finished, the recognition data selectionprocessing is finished, and the processing returns to FIG. 33.

Note that processing of step S1375 is executed in a similar manner asthe processing of step S1135 in FIG. 36.

By performing the recognition data selection processing as describedabove, the electronic shelf label 1101 can supply a result ofrecognition of a customer-specific ID code as recognition data to theserver 1103. Note that the server 1103 can perform processing in asimilar manner as in the case of the second embodiment to select data tobe displayed corresponding to the recognition data, and supply the datato be displayed to the electronic shelf label 1101. Accordingly, theelectronic shelf label 1101 can present more appropriate information toeach customer.

Note that, in FIG. 55, the recognition data selection processing isdescribed to be performed by a method similar to that in the case ofFIG. 36. However, a process of the recognition data setting processingis not limited to this example. For example, the recognition datasetting processing may be performed in a process similar to that in thecases of the flowcharts of FIGS. 37 and 38.

7. Seventh Embodiment

<Mobilization of a Camera>

A shooting direction (angle of view) of the camera 1111 provided on theelectronic shelf label 1101 may be fixed, or may be movable. Forexample, the configuration may be such that the camera 1111 is providedmovably on a housing of the electronic shelf label 1101, and a shootingdirection (angle of view) of the camera 1111 is changeable.

For example, as shown in A of FIG. 55, the camera 1111 is installed on afront surface (surface on a customer side) of a housing of theelectronic shelf label 1101 to pick up an image of the front surface ofthe electronic shelf label 1101. The configuration may be such that adirection of the camera 1111 is made movable, and, for example, thecamera 1111 can be directed to the front surface of the housing (in ahorizontal direction) as in B of FIG. 55, directed somewhat upward as inC of FIG. 55, or directed somewhat downward as in D of FIG. 55.

As has been described with reference to FIG. 29, the electronic shelflabel 1101 is installed on the product shelf 1120. In general, aplurality of the shelves 1121 on which a product is arranged areprovided on the product shelf 1120. That is, the shelves 1121 areprovided in a high location and a low location. The electronic shelflabel 1101, which is installed on the shelf 1121 described above, isalso installed in a high location and a low location other than around alevel of an eyesight of a customer 1601, as shown, for example, in A ofFIG. 56. Accordingly, when a direction of the camera 1111 is fixed onthe electronic shelf label 1101, there is possibility that the cameras1111 of an electronic shelf label 1101-1 installed at a high position,electronic shelf labels 1101-11 and 1101-16 installed at low positions,and the like cannot pick up an image of a face of the customer 1601 fromthe front as shown in B of FIG. 56. Furthermore, in some cases, there ispossibility that, even if the customer 1601 approaches the electronicshelf label 1101, a face of the customer 1601 is not positioned within ashooting range. Therefore, there has been possibility that accuracy ofrecognizing a face of the customer 1601 is lowered. In view of theabove, by making a direction of the camera 1111 movable, an image of aface of the customer 1601 who approaches the electronic shelf label 1101and looks at the electronic shelf label 1101 can be picked up from thefront, regardless of an installation location of the electronic shelflabel 1101.

For example, in a case of the example of FIG. 56, the configuration maybe such that, when each of the electronic shelf label 1101 is installed,a direction of the camera 1111 is adjusted in accordance with a level ofthe installation, so that the camera 1111 is directed to a position of aface of the customer 1601 at a general level.

Note that direction control of the camera 1111 may be performedmanually, or by providing a control mechanism that controls a directionof the camera 1111 in the electronic shelf label 1101. For example, theconfiguration may be such that an image of the customer 1601 is pickedup by moving a direction of the camera 1111 within an entire movablerange so that a face of the customer 1601 is recognized, a range inwhich the face of the customer 1601 can be recognized is found, and adirection of the camera 1111 is matched with a predetermined directionwithin the range.

For example, as shown in A of FIG. 57, the control mechanism firstdirects the camera 1111 most upwardly. Next, as shown in B and C of FIG.57, the control mechanism moves the direction of the camera 1111 in adownward direction from there, and, as shown in D of FIG. 57, a range din which face recognition with respect to a camera angle movable range(a+c) is successful is found. Then, the control mechanism sets adirection (d/2) of a center of the range d to an angle at which a faceof a customer is most easily recognized (optimum recognition angle), anddirects the camera 1111 to the direction.

In the above manner, a direction of the camera 1111 can be set to a moreappropriate direction more easily, and accuracy of recognition of acustomer (face) can be improved.

Note that a movable direction of the camera 1111 is optional. Forexample, the camera 1111 may be movable only in a vertical direction, ormay be movable in left and right directions and an oblique direction.

8. Eighth Embodiment

<Power Supply Control>

For example, in a state in which there is nobody in the vicinity of aproduct (the electronic shelf label 1101), in other words, in a statewhere the electronic shelf label 1101 does not recognize a customer, aproduct, or the like, the electronic shelf label 1101 cannot collectinformation or set (update) information to be displayed. Accordingly, insuch a case, the configuration may be such that part of functions isstopped, and power supply to a processing unit of that part is stopped.In this manner, increase in power consumption can be restricted. Forexample, in a case where a power supply of the electronic shelf label1101 is a battery, a driving time period of the electronic shelf label1101 can be increased by performing power supply control as describedabove.

<Configuration of the Electronic Shelf Label>

A main configuration example of the electronic shelf label 1101 in thiscase is shown in FIG. 58. As shown in FIG. 58, in this case, theelectronic shelf label 1101 has a moving object detection unit 1631, apower supply control unit 1632, a power supply unit 1633, and a powersupply unit 1634, in addition to the configuration of FIG. 30.

The moving object detection unit 1631 detects a moving object in apicked-up image obtained by the image pickup unit 1131. A method of themoving object detection is optional, and may be, for example, a methodsimilar to that of the moving object detection unit 1362 in FIG. 49. Themoving object detection unit 1631 supplies a result of the detection tothe power supply control unit 1632.

The power supply control unit 1632 controls the power supply unit 1633and the power supply unit 1634 on the basis of a result of the movingobject detection performed by the moving object detection unit 1631. Thepower supply unit 1633 supplies power necessary for driving to eachprocessing unit (for example, the image pickup unit 1131, the movingobject detection unit 1631, the power supply control unit 1632, and thedisplay unit 1137) enclosed by an alternate long and short dash line1641. The power supply unit 1634 supplies power necessary for driving toeach processing unit (for example, the image recognition unit 1132, theprocessing control unit 1133, the communication unit 1134, the displaycontrol unit 1135, and the storage unit 1136) enclosed by a dotted line1642.

For example, in a case where a moving object is not detected by themoving object detection unit 1631 (for example, for a predeterminedperiod), the power supply control unit 1632 stops power supply from thepower supply unit 1634. That is, power supply to the image recognitionunit 1132 to the storage unit 1136 is stopped, and processing performedby these processing units is stopped. Furthermore, for example, in acase where a moving object is detected by the moving object detectionunit 1631, the power supply control unit 1632 starts power supply fromthe power supply unit 1634. That is, power supply to the imagerecognition unit 1132 to the storage unit 1136 is started, andprocessing performed by these processing units is started.

That is, in a state where a moving object is not detected, power issupplied only to the image pickup unit 1131, the moving object detectionunit 1631, the power supply control unit 1632, and the display unit1137, and only image pickup of an object, detection of a moving object,display of an image, and power supply control are performed. In thismanner, execution of unnecessary processing is restricted, and increasein power consumption can be restricted.

Note that, in a state where a moving object is not detected, aprocessing unit to which power supply is stopped is optional, and is notlimited to the example of FIG. 58. For example, the configuration may besuch that only power supply to the image recognition unit 1132 isstopped. Furthermore, the configuration may be such that power supply tothe display unit 1137 is stopped, so that image display is stopped.

Furthermore, if power supply to each processing unit is stopped in acase where a moving object is not detected, the configuration may besuch that the power supply is stopped completely, or a small amount ofpower supply is continued (a supply amount is reduced). For example,power supply that is necessary for a processing unit to hold data may becontinued. Furthermore, for example, in a case where power supply to thedisplay unit 1137 is stopped, the configuration may be such that displayof the monitor 1112 is entirely erased, or only brightness of display ofthe monitor 1112 is reduced.

<Linkage of Power Control>

Furthermore, the above power supply control may be linked with the otherelectronic shelf labels 1101. That is, the configuration may be suchthat, on the basis of a result of moving object detection performed bycertain one of the electronic shelf label 1101, power supply to theother electronic shelf labels 1101 is controlled. Note that a method ofthis linkage is optional.

For example, if a customer 1651 moves from the left in the diagram tothe right in front of the product shelf 1120 as shown on an upper sideof A to D of FIG. 59, the electronic shelf label 1101 in a column in thevicinity of the customer 1651 as a moving object (the electronic shelflabel 1101 enclosed by a solid line in the diagram) detects a motion ofthe customer 1651 as shown on a lower side of A to D of FIG. 59. Then,the power supply unit 1634 is turned on not only for the electronicshelf label 1101, but also for the electronic shelf label 1101 (theelectronic shelf label 1101 enclosed by a dotted line in the diagram) ina column on the right (one column ahead in a moving direction of themoving object (the customer 1651)). Then, the power supply unit 1634 ofthe electronic shelf label 1101 in a column by which the customer 1651passes is turned off.

As shown in D of FIG. 59, when the customer 1651 reaches a right end ofthe product shelf 1120, the electronic shelf label 1101 in a column ofthe right end (the electronic shelf label 1101 enclosed by a solid linein the diagram) detects a motion of the customer 1651, and the powersupply unit 1634 of the electronic shelf label 1101 (the electronicshelf label 1101 enclosed by a dotted line in the diagram) is turned on.

As shown in E of FIG. 59, when the customer 1651 changes a movingdirection to a left direction from the right direction in the diagram,the electronic shelf label 1101 in a column in the vicinity of thecustomer 1651 as a moving object (the electronic shelf label 1101enclosed by a solid line in the diagram) detects a motion of thecustomer 1651, and the power supply unit 1634 is turned on not only forthe electronic shelf label 1101, but also for the electronic shelf label1101 (the electronic shelf label 1101 enclosed by a dotted line in thediagram) in one column left of the electronic shelf label 1101 (onecolumn ahead in the moving direction of a moving object (the customer1651)). Then, the power supply unit 1634 of the electronic shelf label1101 in a column by which the customer 1651 passes is turned off.

By controlling power supply in the above manner, increase in powerconsumption can be restricted.

<Process of Power Supply Control Processing>

The electronic shelf label 1101 executes power supply control processingto perform the above power supply control. An example of a process ofthe power supply control processing will be described with reference toa flowchart of FIG. 60.

If the power supply control processing is started, the power supplycontrol unit 1632 sets a power supply to a drive mode in step S1401, andturns on the power supply unit 1633 and the power supply unit 1634. Thatis, the power supply unit 1633 and the power supply unit 1634 are causedto supply power to all processing units.

In step S1402, the image pickup unit 1131 starts image pickup. In stepS1403, the communication unit 1134 acquires data to be displayed fromthe server 1103. In step S1404, the display control unit 1135 uses thedata to be displayed to generate an image to be displayed. In stepS1405, the display unit 1137 causes the monitor 1112 to display theimage to be displayed.

In step S1406, the power supply control unit 1632 sets a power supply toa power saving mode, keeps the power supply unit 1633 turned on, andturns off the power supply unit 1634. That is, the power supply controlunit 1632 stops power supply (power supply to the image recognition unit1132 to the storage unit 1136) by the power supply unit 1634.

In step S1407, the image pickup unit 1131 acquires a picked-up image. Instep S1408, the moving object detection unit 1631 performs moving objectdetection by using the obtained picked-up image. In step S1409, thepower supply control unit 1632 determines whether or not a moving objectis detected by the processing of step S1408. In a case where a movingobject is determined to be detected, the processing proceeds to stepS1410.

In step S1410, the power supply control unit 1632 sets a power supply toa drive mode, and turns on the power supply unit 1633 and the powersupply unit 1634. That is, the power supply unit 1633 and the powersupply unit 1634 are caused to supply power to all processing units.

In step S1411, each processing unit of the electronic shelf label 1101executes information acquisition processing. The information acquisitionprocessing, which is similar to that in the case described in the secondembodiment (FIG. 33), will be omitted from description. In step S1412,each processing unit of the electronic shelf label 1101 executes displaycontrol processing. The display control processing, which is similar tothat in the case described in the second embodiment (FIG. 40), will beomitted from description.

When the display control processing is finished, the processing proceedsto step S1414. Furthermore, in step S1409, in a case where a movingobject is determined not to be detected (for a predetermined period),the processing proceeds to step S1413. In step S1413, the power supplycontrol unit 1632 sets a power supply to a power saving mode, keeps thepower supply unit 1633 turned on, and turns off the power supply unit1634. That is, the power supply control unit 1632 stops power supply(power supply to the image recognition unit 1132 to the storage unit1136) by the power supply unit 1634. When a power supply is set to apower saving mode, the processing proceeds to step S1414.

In step S1414, the electronic shelf label 1101 determines whether or notto finish the power supply control processing. In a case where theprocessing is determined not to be finished, the processing returns tostep S1407, and the processing of step S1407 and subsequent steps isexecuted. Furthermore, in a case where the power supply controlprocessing is determined to be finished in step S1414, the power supplycontrol processing is finished.

By executing the power supply control processing in the above manner,the electronic shelf label 1101 can restrict increase in powerconsumption.

9. Ninth Embodiment

<Control of Face Direction Detection Processing>

Note that, in a case where a plurality of customers are included in apicked-up image, a direction of a face needs to be detected for eachcustomer. That is, the larger the number of customers included in apicked-up image, the larger a load of detecting a direction of a faceincreases. Therefore, the electronic shelf label 1101 may control facedirection detection processing in accordance with the number ofcustomers included in a picked-up image in such a manner as reducing aload.

For example, the configuration may be such that, in a case where thenumber of customers is large, a clock frequency for face directiondetection is increased, or a frequency of face direction detection isreduced.

<Process of Information Acquisition Processing>

An example of a process of the information acquisition processing inthis case will be described with reference to flowcharts of FIGS. 61 and62.

When the information acquisition processing is started, the image pickupunit 1131 starts image pickup in step S1501. In step S1502, eachprocessing unit of the electronic shelf label 1101 performs initialsetting. In step S1503, the image pickup unit 1131 picks up an image ofan object by using the camera 1111 to obtain a picked-up image.

In step S1504, the customer recognition unit 1142 performs facedetection processing. In step S1505, the customer recognition unit 1142determines whether or not the number of detected people is smaller thana predetermined threshold value set in advance. In a case where thenumber is determined to be smaller than the threshold value, theprocessing proceeds to step S1506.

In step S1506, the customer recognition unit 1142 detects a direction ofeach detected face by time division. Furthermore, the customerrecognition unit 1142 similarly performs identification of a detecteddirection of each face (which product a customer faces, in other words,which product a customer looks at) by time division.

In step S1507, the customer recognition unit 1142 uses obtainedinformation associated with a direction of a face and the like tocollect desired customer information. In step S1508, the processingcontrol unit 1133 uses the obtained collection information astransmission information.

In step S1509, the communication unit 1134 transmits the transmissioninformation to the server 1103.

In step S1510, the electronic shelf label 1101 determines whether or notto finish the information acquisition processing. In a case where theprocessing is determined not to be finished, the processing returns tostep S1503, and the processing of step S1503 and subsequent steps isexecuted. Furthermore, when the electronic shelf label 1101 determinesto finish the information acquisition processing in step S1510, theinformation acquisition processing is finished.

Note that, in step S1505, in a case where the number of detected peopleis determined to be larger than or equal to the threshold value, theprocessing proceeds to step S1521 in FIG. 62.

In step S1521, the customer recognition unit 1142 determines whether ornot to increase a clock frequency for detection of a direction of aface. In a case where a clock frequency is determined to be increased,the processing proceeds to step S1522.

In step S1522, the customer recognition unit 1142 increases a systemclock frequency for face direction detection, so as to allow detectionof a direction of a face to be performed at higher speed. That is,directions of faces of more customers can be detected.

In step S1523, the customer recognition unit 1142 uses a system clock atan increased frequency to detect a direction of each face by timedivision. Furthermore, the customer recognition unit 1142 similarlyidentifies a direction of each face by time division.

In step S1524, the customer recognition unit 1142 uses obtainedinformation associated with a direction of a face and the like tocollect desired customer information. In step S1525, the processingcontrol unit 1133 uses the obtained collection information astransmission information. When the processing of step S1525 is finished,the processing returns to step S1509 in FIG. 61.

Furthermore, in step S1521 in FIG. 62, in a case where a clock frequencyfor detection of a direction of a face is determined not to beincreased, the processing proceeds to step S1526. In step S1526, thecustomer recognition unit 1142 lowers a detection frequency fordetection of a direction of a face. For example, operation is changedfrom (1) operation of picking up an image of a customer every 1/30seconds, identifying directions of all faces whose images have beenpicked up within 1/30 seconds, and outputting all of the identifieddirections of the faces every 1/30 seconds, to (2) operation of pickingup an image of a customer every 2 seconds, identifying directions of allfaces whose images have been picked up within 2 seconds, and outputtingall of the identified directions of the faces every 2 seconds. In thismanner, a time period that can be applied to processing of identifying adirection of a face after picking up an image of a customer becomeslonger. Accordingly, directions of faces of more customers can bedetected.

In step S1527, the customer recognition unit 1142 detects a direction ofeach face by time division. Furthermore, the customer recognition unit1142 similarly identifies a direction of each face by time division.

In step S1528, the customer recognition unit 1142 uses obtainedinformation associated with a direction of a face and the like tocollect desired customer information. In step S1529, the processingcontrol unit 1133 uses the obtained collection information andnotification information notifying that a detection frequency has beenlowered as transmission information. When the processing of step S1529is finished, the processing returns to step S1509 in FIG. 61.

By performing the information acquisition processing as described above,detection of a direction of a face can be controlled in accordance withthe detected number of customers, and a failure of detection of adirection of a face can be restricted.

Note that the configuration may be such that the electronic shelf label1101 has a plurality of the customer recognition units 1142, so thatdetection of directions of a plurality of faces and identification ofdirections of the faces can be executed in parallel. Even in such acase, if the number of detected customers is increased, there ispossibility that detection of a direction of a face fails. In view ofthe above, in this case as well, a failure of detection of a directionof a face can be restricted by performing the information acquisitionprocessing as shown in FIGS. 61 and 62.

10. Tenth Embodiment

<Product>

In the above description, the electronic shelf label 1101 is describedto display information associated with a product. This “product” may beany product. For example, this “product” may be tangible, may beintangible like electronic information, or may be a service.Furthermore, a provision form of this “product” is optional, and may beselling, transferring, lending, exhibiting, or copying. Furthermore,these acts may be for a fee or free of charge. In a case where theseacts are for a fee, charging processing may be performed. Furthermore, aconsideration may be paid not only by cash, but also by using a creditcard, and the like. Furthermore, dedicated or generally-used points mayalso be used. Moreover, points may be provided by using the presentelectronic shelf label system.

<Other Systems>

The present technology is not limited to the electronic shelf labelsystem described above, and can be applied, for example, to systems usedfor an optional purpose in an optional field, such as traffic, medicaltreatment, crime prevention, agriculture, livestock industry, miningindustry, beauty, factories, home appliances, weather, naturalsurveillance, and the like, as long as the system controls display ofinformation on the basis of a picked-up image.

<Computer>

A series of pieces of processing described above can be executed byhardware, or by software. In a case where a series of pieces ofprocessing is executed by software, a configuration as a computer thatallows the software to be executed is preferably included. This computerincludes, for example, a computer incorporated into dedicated hardware,a general-purpose computer that can execute an optional function bybeing installed with a variety of programs, and the like.

FIG. 63 is a block diagram showing a main configuration example of acomputer. As shown in FIG. 63, a computer 1900 has a central processingunit (CPU) 1901, a read only memory (ROM) 1902, and a random accessmemory (RAM) 1903 which are mutually connected via a bus 1904.

Furthermore, an input and output interface 1910 is also connected to thebus 1904. An input unit 1911, an output unit 1912, a storage unit 1913,a communication unit 1914, and a drive 1915 are connected to the inputand output interface 1910.

The input unit 1911 has an optional input device such as, for example, akeyboard, a mouse, a touch panel, an image sensor, a microphone, aswitch, an input terminal, and the like. The output unit 1912 has anoptional output device such as, for example, a display, a speaker, anoutput terminal, and the like. The storage unit 1913 has an optionalstorage medium, for example, a non-volatile memory and the like, such asa hard disk, a RAM disk, a solid state drive (SSD), a universal serialbus (USB) memory, and the like. The communication unit 1914 has acommunication interface of an optional communication standard for wiredor radio communication, or both, such as, for example, Ethernet,Bluetooth, USB, high-definition multimedia interface (HDMI), IrDA, andthe like. The drive 1915 drives a removable medium 1921 having anoptional storage medium, such as a magnetic disk, an optical disk, amagneto-optical disk, a semiconductor memory, or the like, that ismounted on itself.

In the computer 1900 configured as described above, for example, the CPU1901 loads a program stored in the storage unit 1913 into the RAM 1903via the input and output interface 1910 and the bus 1904 and executesthe program so as to be able to perform a function equivalent to part orwhole of the configuration of hardware described above in eachembodiment. That is, at least part of a series of pieces of processingdescribed above is performed. Furthermore, data and the like necessaryfor the CPU 1901 to execute a variety of types of processing are storedas appropriate in the RAM 1903.

A program executed by the CPU 1901 can be recorded, for example, in theremovable medium 1921 used as a package medium and the like, andapplied. In such a case, the program can be installed in the storageunit 1913 via the input and output interface 1910 by mounting theremovable medium 1921 on the drive 1915. Furthermore, the program canalso be provided via a wired or radio transmission medium, such as alocal area network, the Internet, or digital satellite broadcasting. Insuch a case, the program can be received by the communication unit 1914and installed in the storage unit 1913. Other than the above, theprogram can also be installed in the ROM 1902 or the storage unit 1913in advance.

Note that part of a series of pieces of processing described above canbe executed by hardware, and the remaining part can be executed bysoftware.

<Others>

An embodiment of the present technology is not limited to the embodimentdescribed above, and a variety of modifications can be made within arange not deviating from the gist of the present technology.

Furthermore, for example, the present technology can be performed as anyconfiguration constituting a device or a system, for example, aprocessor as a system large scale integration (LSI) and the like, amodule using a plurality of processors and the like, a unit using aplurality of modules and the like, a set and the like obtained by addingother functions to a unit (in other words, a configuration of part of adevice).

Note that, in the present description, a system means an assembly of aplurality of constituents (devices, modules (components), and the like),regardless of whether or not all constituents are in the same housing.Accordingly, both of a plurality of devices that are contained inseparate housings and connected via a network, and one device in which aplurality of modules are contained in one housing, are systems.

Furthermore, for example, the configuration described as one device (orprocessing unit) may be divided, and configured as a plurality ofdevices (or processing units). In contrast, the configuration describedabove as a plurality of devices (or processing units) may becollectively configured as one device (or processing unit). Furthermore,as a matter of course, a configuration other than those described abovemay be added to a configuration of each device (or each processingunit). Moreover, as long as a configuration and operation as an entiresystem are substantially the same, part of a configuration of a certaindevice (or processing unit) may be included in a configuration ofanother device (or another processing unit).

Furthermore, for example, the present technology can employ aconfiguration of cloud computing in which one function is shared andprocessed jointly by a plurality of devices via a network.

Furthermore, for example, the program described above can be executed byan optional device. In such a case, the configuration may be such thatthe device has a necessary function (a functional block, and the like)so as to be able to obtain necessary information.

Furthermore, for example, each step described in the above flowchartscan be executed by one device, or shared and executed by a plurality ofdevices. Moreover, in a case where a plurality of pieces of processingare included in one step, a plurality of pieces of the processingincluded in one step can be executed by one device, or shared andexecuted by a plurality of devices.

Note that, in a program executed by a computer, processing of steps thatdescribe the program may be executed in a time series along the orderdescribed in the present embodiment, or may be executed in parallel, orindividually at a necessary timing, such as when the processing iscalled, and the like. That is, as long as no inconsistency is generated,processing of each step may be executed in the order different from theorder described above. Moreover, processing of a step that describes theprogram may be executed in parallel with processing of other programs,or may be executed in combination with processing of other programs.

Note that a plurality of the present technologies described in thepresent description can be performed independently of each other andalone as long as no inconsistency is generated. As a matter of course, aplurality of optional ones of the present technologies can also beperformed in combination. For example, the present technology describedin any of embodiments can be performed in combination with the presenttechnology described in the other embodiments. Furthermore, the optionalpresent technology described above can be performed in combination withother technologies not described above.

Note that the present technology can employ configurations as describedbelow.

(1) An information collection system including an information collectiondevice that collects information associated with a customer or a productfor each product type.

(2) The information collection system according to (1), in which

the number of the information collection devices installed in locationsin which a product is placed is larger than or the same as the number ofproduct types for which the information is collected.

(3) The information collection system according to (1) or (2), in which

the number of the information collection devices installed in locationsin which a product is placed is larger than or the same as the number ofproduct types in the information for each product transferred from theinformation collection device to a data accumulation device thataccumulates the information.

(4) The information collection system according to any of (1) to (3), inwhich

the number of the information collection devices installed in locationsin which a product is placed is larger than or the same as the number ofproduct types for which the information is output.

(5) The information collection system according to any of (1) to (4), inwhich

the information collection device is a network camera that includes animage pickup function and a communication function.

(6) The information collection system according to any of (1) to (5), inwhich

the information collection device is an electronic shelf label thatincludes an image pickup function and a communication function.

(7) The information collection system according to any of (1) to (6), inwhich

the information collection device is an electronic point of purchase(POP) advertising that has an image pickup function and a communicationfunction.

(8) The information collection system according to any of (1) to (7), inwhich

the information collection device is a character information displaydevice that includes an image pickup function and a communicationfunction.

(9) The information collection system according to any of (1) to (8), inwhich

the information collection device acquires and displays informationsupplied from other devices.

(10) The information collection system according to (9), in which

the other devices supply information associated with a product suppliedfrom the information collection device corresponding to identificationinformation of the product to the information collection device, and

the information collection device displays the supplied informationassociated with the product.

(11) An electronic shelf label including an information collection unitthat collects information associated with a customer or a product.

(12) The electronic shelf label according to (11), in which

the information collection unit is an image sensor.

(13) An electronic point of purchase (POP) advertising including aninformation collection unit that collects information associated with acustomer or a product.

(14) The electronic POP advertising according to (13), in which

the information collection unit is an image sensor.

(15) A character information display device including an informationcollection unit that collects information associated with a customer ora product.

(16) The character information display device according to (15), inwhich

the information collection unit is an image sensor.

(21) An information processing device, including:

an image pickup unit that picks up an image of an object;

a recognition unit that recognizes information associated with a productincluded in a picked-up image obtained by the image pickup unit; and

a display unit that displays an image associated with the productcorresponding to the information associated with a product recognized bythe recognition unit.

(22) The information processing device according to (21), in which

the information associated with a product is identification informationof the product.

(23) The information processing device according to (21) or (22), inwhich

the image associated with a product includes at least any one of aproduct name, a price, and identification information of the product.

(24) The information processing device according to any of (21) to (23),further including:

a communication unit that performs communication with other informationprocessing devices, supplies the information associated with a productrecognized by the recognition unit to the other information processingdevices, and acquires data to be displayed corresponding to theinformation associated with a product; and

a display control unit that uses the data to be displayed acquired bythe communication unit to generate an image associated with the productthat is to be displayed on the display unit, in which

the display unit is configured to display the image associated with aproduct generated by the display control unit.

(25) The information processing device according to any of (21) to (24),in which

the image pickup unit picks up an image of a customer as an object,

the recognition unit recognizes the information associated with acustomer on the basis of a picked-up image obtained by the image pickupunit, and

the communication unit supplies the information associated with acustomer recognized by the recognition unit to the other informationprocessing devices.

(26) The information processing device according to any of (21) to (25),in which

the information associated with a customer includes at least any one ofa position of a face of the customer, a time at which the face of thecustomer is detected, a time period in which the face of the customer isdetected.

(27) The information processing device according to any of (21) to (26),in which

the information associated with a customer further includes attributesof the customer.

(28) The information processing device according to any of (21) to (27),in which

the information associated with a customer further includesidentification information of the customer.

(29) The information processing device according to any of (21) to (28),further including:

a motion image pickup unit that picks up an image of a motion of anobject; and

a detection unit that detects the motion of the customer on the basis ofa picked-up image obtained by the motion image pickup unit, in which

the communication unit is configured to further supply the motion of thecustomer detected by the motion detection unit as information associatedwith the customer to the other information processing devices.

(30) The information processing device according to any of (21) to (29),in which

the recognition unit identifies a position of the customer included inthe picked-up image, and recognizes the information associated with thecustomer only for the customer positioned within a predetermined range.

(31) The information processing device according to any of (21) to (30),in which

the communication unit acquires data to be displayed corresponding tothe information associated with a customer supplied to the otherinformation processing devices, and

the display control unit uses the data to be displayed acquired by thecommunication unit to generate the image associated with a product to bedisplayed on the display unit.

(32) The information processing device according to any of (21) to (31),in which

the communication unit further includes a control unit that controlswhether to supply the information associated with a product or to supplythe information associated with a customer.

(33) The information processing device according to any of (21) to (32),further including a specific code recognition unit that recognizes aspecific code included in a picked-up image obtained by the image-pickupunit, the information processing device in which

the control unit is configured to control the communication unit tosupply the information associated with a product or the informationassociated with a customer in accordance with a result of recognition ofthe specific code performed by the specific code recognition unit.

(34) The information processing device according to any of (21) to (33),in which

the control unit causes the communication unit to supply the informationassociated with a product in a case where a first specific code isidentified by the specific code recognition unit, and causes thecommunication unit to supply the information associated with a customerin a case where a second specific code different from the first code isidentified by the specific code recognition unit.

(35) The information processing device according to any of (21) to (34),in which

in a case where a specific code is recognized by the specific coderecognition unit, the control unit causes the communication unit toswitch information to be supplied.

(36) The information processing device according to any of (21) to (35),in which

the control unit causes the communication unit to supply any one of theinformation associated with a product and the information associatedwith a customer in a case where a specific code is not recognized by thespecific code recognition unit, causes information to be supplied by thecommunication unit to the other one in a case where a specific code isrecognized by the specific code recognition unit, and returnsinformation to be supplied by the communication unit to the first one ina case where a predetermined period of time elapses since the specificcode is recognized by the specific code recognition unit.

(37) The information processing device according to any of (21) to (36),further including:

a moving object detection unit that detects a moving object on the basisof a picked-up image obtained by the image pickup unit; and

a power supply control unit that turns on a power supply of at least therecognition unit in a case where a moving object is not detected by themoving object detection unit for a predetermined period or longer, andturns on a power supply of all processing units in a case where a movingobject is detected by the moving object detection unit.

(38) The information processing device according to any of (21) to (37),further including a customer recognition unit that detects a face of acustomer included in a picked-up image obtained by the image pickup unitand identifies a direction of the detected face, in which

the communication unit is configured to supply information showing thedirection of the face identified by the customer recognition unit, andcauses an image for the customer to be displayed on an electronic shelflabel arranged at a position corresponding to the direction of the face.

(39) An information processing method, including:

picking up an image of an object;

recognizing information associated with a product included in anobtained picked-up image; and

displaying an image associated with the product corresponding to therecognized information associated with a product.

(40) An electronic shelf label that displays an image associated with aproduct, the electronic shelf label including:

an image pickup unit that picks up an image of an object;

a recognition unit that recognizes information included in a picked-upimage obtained by the image pickup unit; and

a display unit that displays an image associated with the productcorresponding to the information recognized by the recognition unit.

REFERENCE SIGNS LIST

-   100 Information collection system-   101 Shelf device-   102 Relay device-   103 Information management server-   121 Product display shelf-   122 Information collection device-   123 Local server-   131 Product-   140 Network camera-   141 Image sensor-   142 Ranging sensor-   150 Electronic shelf label-   151 Image sensor-   152 Ranging sensor-   153 Monitor-   160 Electronic POP advertising-   161 Image sensor-   162 Ranging sensor-   163 Monitor-   164 Speaker-   411 Information provision server-   1100 Electronic shelf label system-   1101 Electronic shelf label-   1102 Network-   1103 Server-   1111 Camera-   1112 Monitor-   1113 Mode identification LED-   1131 Image pickup unit-   1132 Image recognition unit-   1133 Processing control unit-   1134 Communication unit-   1135 Display control unit-   1136 Storage unit-   1137 Display unit-   1141 Specific code recognition unit-   1142 Customer recognition unit-   1143 Product recognition unit-   1341 Camera-   1361 Image pickup unit-   1362 Moving object detection unit-   1363 Processing control unit-   1471 Customer unique ID code recognition unit-   1631 Moving object detection unit-   1632 Power supply control unit-   1633 Power supply unit-   1634 Power supply unit

1. An information collection system comprising: an informationcollection device that includes an image pickup unit for each producttype; and an image analysis device configured to compare two or moreimages of a person picked up by a plurality of the image pickup unitsand identify information associated with a person.
 2. The informationcollection system according to claim 1, wherein the informationassociated with a person includes information associated with whichproduct type a person whose image is picked up looks at.
 3. Theinformation collection system according to claim 2, wherein theinformation associated with which product type the person whose image ispicked up looks at includes information associated with a result ofidentification of the person who looks at the product type for eachproduct type.
 4. The information collection system according to claim 3,wherein the information associated with which product type the personwhose image is picked up looks at further includes informationassociated with a number of people who look at the product type within acertain period of time.
 5. The information collection system accordingto claim 3, wherein the information associated with the result ofidentification of the person who looks at the product type includesinformation associated with a time length in which the person looks atthe product type.
 6. The information collection system according toclaim 3, wherein the information collection device is a network camerathat has an image pickup function and a communication function, anelectronic shelf label that has an image pickup function and acommunication function or an electronic point of purchase (POP)advertising that has an image pickup function and a communicationfunction.
 7. The information collection system according to claim 6,wherein the comparison of the two or more images of the person iscomparison of directions of a face of the person in a plurality of theimages.
 8. The information collection system according to claim 7,wherein the information associated with the result of identification ofthe person who looks at the product type identifies a product typecorresponding to the information collection device that picks up animage of a face that faces front the most on a basis of comparison ofdirections of a face of a person in a plurality of the images.
 9. Theinformation collection system according to claim 8, wherein thecomparison of directions of the face of the person in a plurality of theimages is comparison, between images, of size of two sections in a faceof a person included in a plurality of the images.
 10. The informationcollection system according to claim 9, wherein the comparison of sizeof two sections between images is comparison, between images, of ratiosof a width of a face region to a distance between a center of the faceregion and a center of a face.
 11. The information collection systemaccording to claim 9, wherein the comparison of size of two sectionsbetween images is a ratio or an absolute value of a difference between adistance between a left end of a face region and a center of a face anda distance between a right end of the face region and the center of theface.
 12. The information collection system according to claim 9,wherein the comparison of size of two sections between images is a ratioor an absolute value of a difference between a distance between a centerof a face and a center of a right eye and a distance between the centerof the face and a center of a left eye.
 13. The information collectionsystem according to claim 8, wherein the information collection devicefurther includes a ranging sensor, and supplies a result of distancemeasurement obtained by using the ranging sensor to the image analysisdevice.
 14. The information collection system according to claim 13,wherein the image analysis device identifies same person on a basis ofthe result of distance measurement.
 15. The information collectionsystem according to claim 8, wherein the image analysis device isconfigured to detect information associated with a motion of a person bycomparing the two or more images of the person, and identify same personon a basis of the detected information associated with the motion of theperson.
 16. The information collection system according to claim 15,wherein the information associated with the motion of the personincludes at least any one or more of a movement start time, a movementstop time, a time at which the person moves into a frame, and a movingdirection.
 17. The information collection system according to claim 6,further comprising: a communication device that transmits informationacquired from the information collection device and a result of imageanalysis performed by the image analysis unit.
 18. An electronic shelflabel, comprising: an information collection unit that includes an imagepickup unit for each product type; and an image analysis unit configuredto compare two or more images of a person picked up by a plurality ofthe image pickup units and identify information associated with aperson.
 19. An electronic point of purchase (POP) advertising,comprising: an information collection unit that includes an image pickupunit for each product type; and an image analysis unit configured tocompare two or more images of a person picked up by a plurality of theimage pickup units and identify information associated with a person.20. A character information display device, comprising: an informationcollection unit that includes an image pickup unit for each producttype; and an image analysis unit configured to compare two or moreimages of a person picked up by a plurality of the image pickup unitsand identify information associated with a person.